{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Network Optimization and Tuning\n",
    "\n",
    "You've learned how to build computational graphs in PyTorch and compute gradients. The final piece to training a network is applying the gradients to update the network parameters. In this tutorial you will learn how to implement a number of optimization techniques in PyTorch along with other tuning methods. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torchvision import transforms, datasets, models\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from torch.autograd import Variable\n",
    "from collections import namedtuple\n",
    "from IPython.display import Image\n",
    "%matplotlib inline\n",
    "np.random.seed(2018)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the PyTorch dataset API to load a dataset with exactly the same properties as the MNIST handwritten digits dataset. However, instead of handwritten digits, this dataset contains images of 10 different **common clothing items**, hence the name **Fashion-MNIST** . Performance on MNIST saturates quickly with simple network architectures and optimization methods. This dataset is more difficult than MNIST and is useful to demonstrate the relative improvements of different optimization methods. \n",
    "\n",
    "Some of the characteristics are mentioned below.\n",
    "\n",
    "- 28x28 images\n",
    "- 10 classes\n",
    "- Single color channel (B&W)\n",
    "- Centered objects\n",
    "- 50000 training set members\n",
    "- 10000 test set members\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fashion Class that enables the Dataset download and basic transformations\n",
    "class Fashion(datasets.MNIST):\n",
    "    def __init__(self, root, train=True, transform=None, target_transform=None, download=False):\n",
    "        self.urls = [\n",
    "            'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz',\n",
    "            'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz',\n",
    "            'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz',\n",
    "            'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz',\n",
    "        ]\n",
    "        super(Fashion, self).__init__(\n",
    "            root, train=train, transform=transform, target_transform=target_transform, download=download\n",
    "        ) \n",
    "\n",
    "def decode_label(l):\n",
    "    return [\"Top\",\n",
    "     \"Trouser\",\n",
    "     \"Pullover\",\n",
    "     \"Dress\",\n",
    "     \"Coat\",\n",
    "     \"Sandal\",\n",
    "     \"Shirt\",\n",
    "     \"Sneaker\",\n",
    "     \"Bag\",\n",
    "     \"Ankle boot\"\n",
    "    ][l]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = Fashion('/tmp/data', train=True, download=True, transform=transforms.Compose([\n",
    "                                    transforms.ToTensor(),\n",
    "                                    transforms.Normalize((0.1307,), (0.3081,))\n",
    "                            ]))\n",
    "\n",
    "test_data = Fashion('/tmp/data', train=False, download=True, transform=transforms.Compose([\n",
    "                                   transforms.ToTensor(),\n",
    "                                   transforms.Normalize((0.1307,), (0.3081,))\n",
    "                            ]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random examples from the Fashion-MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAE8CAYAAADkCzT6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcZVdZ7//vc4Y6NXZVz1N6SGdOSIBADEOIgsyjKCoGRRRUFEcQB/Bymbx4EZkUAQVRgwQCIpOCyAUiISFAYibS6aSTnuepqqtrPMP6/VEnP+pZa1Onund1dXXV5/168SJr11777FNnnV2r9/meZ1kIQQAAADg1hTN9AgAAAGczJlMAAAA5MJkCAADIgckUAABADkymAAAAcmAyBQAAkAOTKeAsZmbBzM6fxn4bm/uWZuO8AGAhYTIlycxOTPpfw8xGJrVfdqbPD2cfM7vGzG4xswEzO2pm3zazq870eWFu41qE2WZm15nZ95tjbJ+ZfdnMrsl5zG+a2atm6hzPBvwrVVIIofuR/zaz7ZJeFUL42o/a38xKIYTabJzbXD4HZDOzRZK+JOk3Jd0oqU3SUySNncnzwtzHtQizycxeK+lPJL1a0n9KGpf0bEkvknTzGTy1sw53pqbBzN5uZp8ysxvMbFDSL5pZu5m9vzmT32Nm7zaztub+rzKzb07qX2p+xLKx2X6+mW02s0Ez221mfzBp3xea2V1m1m9mN5vZoyb9bLeZvd7M7pE0PEtPHyfvQkkKIdwQQqiHEEZCCF8NIdxtZueZ2dfN7IiZHTazfzGzvkc6mtl2M/tDM7u7eVfrU2bWPunnr2+Oub1m9quTH9TMnmdm/2Nmx81sl5m9edaeMWYF1yLMFDPrlfRWSa8JIXw2hDAUQqiGEL4YQni9mVXM7L3Na83e5n9Xmn0Xm9mXzOyQmR1r/vc5zZ/9uSb+8fg3zbtdf3PmnuXsYTI1fS+W9AlJvZI+JelNkh4v6QpJj5X0ZEl/Os1jfUzSK0MIPc3+N0lS82Ogv5f0KklLJf2DpM8/cmFseqmk5zTPA3PTA5LqZvZPZvYcM1s86Wcm6R2S1ki6RNI6SW+O+v+cJv51eK4mxscrJMnMni3pDyU9Q9IFkp4e9RuS9HJJfZKeJ+k3zeynZuxZYa7gWoSZ8ERJ7ZL+7Uf8/I2SniDpMZIeLenHJP1Z82cFTYydDZLWSxqR9DeSFEJ4o6RvSfrtEEJ3COG3T9cTmEuYTE3fzc0ZeyOEMCLpZZLeHEI4FEI4qIkZ/i9N81hVSZeaWU8I4WgI4Y7m9l+X9LchhO8172j8Q3P75KzN+0IIu5vngDkohHBc0jWSgib+IB0ysy+Y2coQwtYQwn+FEMZCCIckvVvSj0eHeH8IYW8I4aikL2riYiZNTLI+FkK4N4QwpGgSFkL4ZgjhnuYYvVvSDRnHxtmPaxFmwlJJh6f4iPZlkt4aQjjYvFa9Rc1xFUI4EkL41xDCcAhhUNKfa4Ffa5hMTd+uqL1a0o5J7R2S1k7zWC+W9EJJO5tBvaub2zdI+uPmbfV+M+tvPs7k48bngTkohLA5hPCKEMI5kh6liTtR7zWzFWb2yebHMcclfVzSsqj7/kn/PSzpkRzNGvnXf/L4k5ldbWbfaN56H9BEDiI+Ns5+XIswE45IWmY/+hu+a5SOqzWSZGadZvZhM9vRvI79t6Q+Myue1jOew5hMTV+I2vs0ccF5xHpJe5r/PSSpc9LPVrkDhXBbCOGFklZoIqj8yeaPdkl6Swihb9L/OkMIN05xHpjjQgj3S/pHTUyq3qGJ1/CKEMIiSb+oiY/+pmOfJj4WfMT66OefkPQFSetCCL2SPnQSx8bZg2sRZsKtkkYl/agowF6l42pv879fJ+kiSVc3r2PXNrc/cr1ZcGODydSpu0HSm8xsmZktl/S/NHGXQZLuknSFmV1uZh2S/vcjncyswya+iroohFCVNCip3vzx30l6jZldZRO6zewFZtY1e08LeZnZxWb2ukmBzHWSfkHSdyT1SDohqd/M1kp6/Ukc+kZJrzCzS82sU5PGVVOPpKMhhFEz+zFJ1+V9LjgrcC3CSQshDGgib/cBM/up5t2mcjPn+U5NjKs/M7PlZrasue8j46pHEzmpfjNbovRadEDSptl5JnMDk6lT9xZNXKjukXS3pNs0cddBIYT7JP0fSd+UtEUTt0An+2VJj9wefaV++Dn0bZr4Ov0HJR3TRJD5F0/z88DMG5R0taTbzGxIE5OoezXxr7m3SLpS0oCkf5f02ekeNITwZUnvlfR1SVub/z/Zb0l6q018y+tNmph8Yf7jWoRTEkJ4t6TXaiJYfkgTdyR/W9LnJL1d0vc1MabukXRHc5s0cR3qkHRYE9e3r0SHfp+klzS/6ff+0/w05gQLYcHdjQMAAJgx3JkCAADIgckUAABADkymAAAAcmAyBQAAkMOsLnT8jMLPzpm0+9b3PsFvWJquQVva2e7ajei3VcxYtrZQ9WV96m3+KVvGb2DsnKprL1854NrtpbRA7YmxNtde/sIt6YHPkP9qfPq01jaaS+MIp8/pHEdzaQwVenpcuzE4eIbO5NRY2V+LQnX8DJ1JimvRWawQ1f9s1NN9LHp5T9MX6qYzjrgzBQAAkAOTKQAAgByYTAEAAOQwq5mpueSfX/i3rv21wUcl+7zkKbe79lDwv64uS7NM7eY/1x0N/nPfQ/V0NYYlxWHXrgY/x33H7ucmfX5roz+3j7kllACcLeKMVKG9PdnHzl3n2o0Ht0U7tP53cWFRt2tbZ2eyT+jucO3xlT7P1XbwRNKnft8DLR8bOU0jG1To8n9bDv/cFck+R5826tpPPM+Po52Di5M+F/UddO2RerllnyXt/m/aPXec69rnfXok6WO33OU3RBkpK6XTlVBL/wafKdyZAgAAyIHJFAAAQA5MpgAAAHJgMgUAAJDDggiglzZtTLZdUP62a394eFmyz2sP/6xrV6LimWO19Ne344gP440P+4J2pXZfoFOS+np8GC8u0rmiMy3it6Tog6DFRYtcu378eNIHZ5E4cBo7TcXppmP/5y5x7eXv60j2KX7jDtcuRGHnxrAPqOKHdnz8/GTb3z72E659/9hq1y5kVAM+VvOB5M7CAdeuFNJrUSP4cbepzYePi9ZI+rzzvMuTbcjHKhXXDmO+QnTp3PQLRz//Zf83bUXp+8k+b9v6PNe+drH/8kC1LyqUKWnL8CrX7iv79+5TF9+f9DlQ7fXti/2XH578wc1Jn89t94H5FS/yx80Mm89S0c7p4M4UAABADkymAAAAcmAyBQAAkMOCyEwdu2pVsu07o8td+weH031WdvtcUl+b/6x49aI0l/S4JTtduztaDflYLS2Ut21oqWsfHvGfL+8YWJL0eXCpP9+w6Ry/w533JX1wGpzKZ/bT6RNvi/tkZapaPHacw5DSLEZ48mNc++c/8pWkzyt773Ttp77hRck+xW9EGxpp1gbZfuq8u5NtNw9d6Nr1qLDvktJQ0uep3f4asC66Fn3+xEVJn4dGV7h2nH05NO6LeE7IWPEd05fxXo7fl7GHf2ltsu0zBx7v2vfek+aqCiN+3Kw4r3W29raqL7j59MV+XL180eGkz58d9H9f9+/0f8P++cjVSZ/nXXyva2+9YJNr1x98OD25M5iRinFnCgAAIAcmUwAAADkwmQIAAMhhQWSmhlalc8Z7R/zCocVC+tlrZ2nctUejBR7jBR+zxBmDrD6D435h0/aSr/9yfDTNusQ1ZAbP84/T5WMtOF1aZZum0SdrAc9E0dd/sba2ZJd4wVwVfJ+sHMbIi37Mtd//3r927eMhHXsf6vd5jY7fSmvT1KN2o0UGBD90bc+WZNtdI+tde/+YzzLd3u9/LknP3fgD1/5UtJj7wXFfm06Sjtfap2z/zdqbkz7P1+OSbTgJ08j9bPuLJ7p2tS+tuXTP5mgMFNPjVo7669NbN/u6U//vsR9L+vSt8K/5prLPWR2sp9e8f7vxKa5dWBZlJkfSa8aXvnOl3/BG/xz7bvO/A0la/sFbk21nCnemAAAAcmAyBQAAkAOTKQAAgByYTAEAAOSwIALoY0vTIN5A3S/O2laMI7PScC1apLjgQ3SDUThTki7o9AuDbhvxCyivrLQuknYoKtppGYuY7h3rc+2B83ygz8fTMWtOoYhc5gKesWifVkX9JEkNP6aLF6UL6H7ir9/t2g/X/Nhrt3Qx3H/88xe4du+D30kfew4tQHq26Suki0APN/y1KF5sdnN1ZdLnWyO+6OFYw3/5pVJIx12t4a8jx8b9dbKhtPhq8dKooOh9DyT74OTEv9PqUv9adWxPv8g0tty/No2e9PWtPPGIa/dvW+zaN13ii7ZK0k91+eLV1eDHxOv3pwU4xy4bce3Hrt/l2v9ze3otajvix17HZt8eeLI/piSt/qL/Mkxt955kn9nCnSkAAIAcmEwBAADkwGQKAAAghwWRmRpfnH7OP1TzxQgrpfTz5TgjVZDPfQxW04KGDfmsSC1akHRRaTTps2vEf27dERXtXNaZZiiOV31ea2wJmZQ5oZAWo4uzS9NR2ugL8NVW+CKNY8vTvN6Bx/scxdgK/7gho4jfXeM+0/ffgxe79oXt+5M+S2/2uYRpJL4whUKXTzheVUmLIH5hwI+rsYa/dFeK6auwY8y/tnEB4eO19Pp1Tnv/lOd613i6bXiDH5sV1ljP7eCTlrp2m4/iqpTGh1R50P+tGTonzVXVlvp92gb8WPvMIb9YsiStK33VtffUfF73rqPposvFbf76dHi5z2J27m59HyeqS63izoxr3rP9dXLpR8hMAQAAnJWYTAEAAOTAZAoAACCHBZGZUka9jYNj/jPcaj3NuhTKPmMyUvOfQR8f98eQpMsX7XXt1ZWBlqdXa/g5bfw4Q9V0Udveij/fRpnM1Fxg5fQtFcZ8dqnw6Etcu/HuaIFiSef0HHLtPcM+R/eatTclfb42cJlr/97yb7j2rz94XdLnvwb84re9URjjaD2tWBYynuNMmNaCz/OQrfa1fcZCer164ITfZ01H6+tKvEhxrH+8M9m2vuOYaw/V/bXnpiGfqZOksT5/LUqTWDhZY31T12kLGW+V6SQzRx7weaclW/3P79vrr02S9LKnrfPndsyPqyV3pCfTG0WDdyzz47cnjTGrFg3HJBeWsYb88Cq/cWm6y6zhzhQAAEAOTKYAAAByYDIFAACQA5MpAACAHBZE4tNKadqtZH5bOWOh40MjPnzbW/GpunIhPe72ER+Be/6Su1x761i6IGkrcfFQSWoEH7xrLIhXcu6bzgLEjbs2u3bp5WuSfbbvidOXvv0BXaiUD6n/lq5x7Xdu+0zSY3nRV2F818GnuvYN/3Ft0ufcrbe6dlZwPNTrU+9j6b/jQjWjIuQCUFu+yLUP1NPfQ2dp6t9NezFdkDpWKvjXJL6GSFIxui72j/lFbfeM+QLDkjSyLCpM3PJM0Eocxo6FjNsg8RDJ2id+yYdW+w2WEQyvb/dftGqLhlq1Ox1H473RhkpcQDi9Zlj0Jzg+16y/caMrT74g8unCnSkAAIAcmEwBAADkwGQKAAAghwWRtCnsTovXdV3gP2CuN05+Xpm1OPLOEz5TsHHlYdfePJrmYxrRh9vxQsdj9fRl6in7bE7HAebFZ6vanr3pxmjB5LgY6HSyWbHXbE6Ldt706Btce+vgcte+8Inbkz5xOifUWi91PJ19SmvT98aCkFGMMNZX9pm5kXpayDe2uOwXSO8s+Gve/tE03VSOgivVhh+H8TEkqdH6VHCSGhVfpLPW6dv19rRIc+WoH0i1njRPVByKritRRiozM7XSX2tqI/5aVD6ekX+KT28aGd8ovqnh1f5k6p3pyVknmSkAAIB5gckUAABADkymAAAAcmAyBQAAkMOCCKB37UkTnsvbBl37u8Prk32WdQ9NedxyIQ2/xQU2H1fx6cwPHUzXtW5ECdSRWnnKx5WkehToK019qpjLLCOBHPw4mk7g3Mp+rMVFMMMNPlwuSZXH+LEWj9+XrPx+0ueGHr+yfGNwMNkn8YQrXHPle7Ynu9x1IK70tzA0Kj4UXM1IpI9Fid19Iz48vqjsCwpL0gNDK1z7lm2bXPtll30v6bNteJk/t+g6Uw3+XKW02CLyq/b492Fh3L8OhXP8lwskqbbbF9eMw+aSVDnmj1OPvpvV1p+eS6hPfc+lkFEvtuZrvap4xF9nGpWMx4nr+taj90ExDd1rcO5MYbgzBQAAkAOTKQAAgByYTAEAAOQwdz5wPI3aj6XFvrqLPoNSzFhMOM5EDVd9JmV9z9Gkz7cfOs+139J3qWuf33kw6fPwoM9RxQU5h5RWxVtU8vtkFVvDWSJkZAFO5TD1qcMrfdffmmy7+20+a7Ox64hrPzC6Oulz7EWXuXbPjjTP9cqPfC7assO1Lq+khUr/6GW/4De8MNllXhpf5C/D/dOoglmt+zxMV0daTHOg6gMxhaiS4tq2Y0mfB0/4nNVozZ9bnN2SpIwYFXIK3f69XNgf5WgzFqmOc0ptx9N94nxbPNSyFkdWfHlKKnJmiI4TFxTNGjPxto6Dvs/g0ozH7WldDHi2cGcKAAAgByZTAAAAOTCZAgAAyGFBZKYq/a0DRZbxOXCIPpceGvcfMF+1yOdAJGnnJy5y7Rsf/ROu/W+v/sukz+d2+xo8cVbr6HD0Ybikao+fBw+dMzO5G8yCuK7UdDJTcR/L+HdQOPng3JcHL3ftTR2HXPvy9l1Jn7e/8x7Xrmc87neiGNVgw4/h39ySLrrc8fC2Kc91vqq1+9dysJEuzF6LFhwej9qVQpod2T/ka1HV96fXkVhPRr2qybIyU7X0dHESCl1dybb2Hv8GqkeZqUYjff/HZQ8zy0PFMap4oeOsS1Ejvva07hMPk+I0ok1RjFljS/yBbTQNWnWsPtH6wLOEO1MAAAA5MJkCAADIgckUAABADkymAAAAclgQAfSu+w8l29qj1RnjsLkknYiKdDaioN2SUhp+69w+4PepLHbtlcV0/lqLCvDVo8pp8c+lNHC69B4C6GeNmSjS2ZiZ1WW/frkPvz79Xr9o8U92pI9z5Vt/07Wri9L3zl//xodce13Jr6B65JtpMdBztDAD6PW2jIWuW4iLaRYyqvaOVH1oua3fX1f2jfed9ONmaVS49uRRWLI42dZW9tf3kejnxVL6vhxf5F+H4lg6ruI/JfFrF4rTGItlP9YaWQU4S/64jXiMZzxM/Pe1uiha7L2cjvGeDp9aL7Sn34ZojE79pYqZwp0pAACAHJhMAQAA5MBkCgAAIIcFkZmqb02zGIer3a7d0VZN9ilHix83os+pV0U5EEkKD+907UUj/jPdbqskfbra/CKljSi/1Zbx+Xg5WrGy71N3+PNIeuCs0qKwp5XTxXBDLRrDcTYrPqakf93lFz/eWvX7PGvN1Umf5UoXTI71/1qna7ebP7eN16cFb+fOkqWzK17gdWlhuHWf6BqxqJTmQsZq/sCFaC3knmLap2w9rh3nNeMF4iWp3npdZkwhdKY5n2Jh6gK8y3rTvO6Bdv+eKw1lLHQctWvdUTujAGu52w+c6lBUQDRag1lKx3S90jrPlSy63OZ/B+XedOy1l/xVw7o6k31EZgoAAGDuYzIFAACQA5MpAACAHBZEZirL9uGlrt1ZTjNTHSW/baTmPxzeX0vrtMQ1LRrbfDakkZFmWtHpa/uM1v3j1DMWtYwzEqFKcCG3KFNkxeiD/6zFhSOhnlH/aYZqQrV8nBb1q676nzSV9PKHX+DaQ9emNdlayartEmek/m3gca5d273npB9nvorr9FQsfW0L0Wqy8TVhWdlfQyTpRL/Pjyzd74/RW0yzWSPRtSdrAfhYKJPQzCNU0j/D3RWfDxqMLj0HjvpFrCWpONK6llMck6sc9TsV0j+DGo/yeRbVkMoaIqVBm3KfrMeJs1dW8096yaJ0vMb11rp6upN9dORouu004M4UAABADkymAAAAcmAyBQAAkAOTKQAAgBwWbAB9x6BfXLK7PJ7sc2TEBzjjPN/mkTUn/bg/GE9DwLUoTDoeFcorF9NA6vaRpdGWNIC6YGQUo2xZsDIrrB1tC7U5XEZyGqH2oa9scu3Pbk0L2q17yb1TH6SQsYpp9NjWln75YU3JL/j92c2Pce1NunPqx13ABkNaBTEu5Ftv+PbGtsNJn9I+/7osvt8HeHuK8fK5UjVaZH1JR+sCoo0F+1dkZoRy+h6Lv/yULBxcS++DxPWgbSh9rNEV0RcZomKaXef6960klar+BS63+evi6OVpgdHiNv+llFq3f5zSiaxrdtSOLnFrutNze+joMn+Irozr1SzhzhQAAEAOTKYAAAByYDIFAACQw4L9tHvvPp+ZevSm3ck+J6JCmJVoUcWxjLBAod1/cB0X8TzSSHMrpWhRy9FaxsqRkYHxuFDiAs5MZeWfppOROkl21eXJti2v7HDtS/98b7JPbVc6tpxp5JIKXV3+x0NpIOLBv/aLEj9jyd2uvf3ZaUampWlks7IKiLZHhSdte0eyTyIr+7YA1Cv+eTdC+m/coWg14bEox9Jp6SKwnfv8cct7ffHCjeU0ZzVc849TjSqKnqinC7WH8tSL8mJqoZjxekd/e+pd/ndsjfS9krFudaLtmO9X85cV2U3+76IkjT02KhAd5bW6tqSZybHFUWHPKH5qGUOmHOWoqot8e03H8aTPvVFuudEx8wWSp4s7UwAAADkwmQIAAMiByRQAAEAOCzYzpRGfBShkfYgbKUX7VAppHaLG6NTZnD3V9DPpWFx3qiNjEebdg36R5cU6+QVqz1ZW8bkNy8jahLp/rULV1xGL80WSdP3zPujaN524JNrju0mfD/Te7trf/PHzk31uvGRVss3JyiVFzynOSBUvSh/njU//vGt/5qVPjfbYnPQp9PT4xxmMsnfTyXOtXJ7sEtcrWnPzNGp2TWMh6fmoEcWQsq5FhagIT7Ho97m47VjSpzzo+9R27HLtHkuvK/GCynF9q45ixgq1ZzCnMh/UutPM0cBIlDns8u8fK6R/Zwpj/rXKiN4p/pMV71NL1yyX4nxWtfXiyG3Ho33Go6xWGh1WIS71GI/FjJWb63X/BGrd6fVqtipPLcyrFwAAwAxhMgUAAJADkykAAIAcmEwBAADksGAD6J27/FNvf3QakI3jbrUorXe0GlU8kyRNXTntByPnJNu6Sj55F0cLixmB1P5BXwSxdax9/ghjvkDhqZTjvPhRu5JtT273r29dW1y7LV55U9K3Rza69hM6tiX7/N3LX+zaff98a+sTbFFkdOPH00Kgb//O81z7wrtuT/aJJYHzUzC2fkmybU9tkWtX/uN7uR9nvooy3pnB8PgaEVtf6k629T089bXoweqyZNviNr+w8ZFRf40rW/oesGL+grgLWbUnjUgPDfjre3u3v+aNHEoT3PF3A7IC6PWoHnSIHrqYMcws+rJDKETXybSOa3LcOHBeTGvMJhfy0NZ6XDUG/ROqpm8DAugAAABnAyZTAAAAOTCZAgAAyGHBZqb6tkaFBy39fHa87j9tjRckjgvpTcf3j6xPtl3Wt2/KPm3FNKdQPZ7xQfUCMfKiH3Pt4+vTYbz6H+5y7bjo5ZOWPtzyce4f84to3ju0Ntnn8Jj/kH53z9Jknz944ydd+2P/vKHlY8dqX/Pj5neWfzLZZ8ef+ezSNMpkzoixxenC3HtrJ5/is8LCXOg41p6RkYwLBHe2+YDMcCMNu5R/sNO146vI3SPptWhNZcC1t1s6njGz6m3puLdj/j1VWeyvXyNZRTujIZCVZSpEgyAuphlnnaSMWrpxtiljFhEfJ44BZp5btM+iVa3znOUB/0D1yplbdJs7UwAAADkwmQIAAMiByRQAAEAOCzYz1b1rxLWHa2nuo1iY+vPX47Ws3FJWAY0f2j/Qk2y7pPfAlH2yFI/PVvWMuWdgkx+2//26v0r2+drvrHTtHeO+ps7Tu+9L+uys+VzKibpf9fP5fXcmfZ7Z6T/oHwtpjaCK+bH1px/6Wde+6O99HkKSRt/ht33sgo+79i9tfnnSp2tP6xzY6XBidToWt46uzNhzaqGxMOsVxbV92jOiY/G1ZkXXCdcezhh3jf6BZNtkDwytSLad097v2uUor9mZVYgoXggXJydjofY4lzQ86hdDrixOa4gVx/x1sZ4xkOrRmsr1dv9AcYZKkurj/p5LocNfJ4uj6d/OsSX+uKVatHB7JX2vt0XR4d5ufw2sNtLrTLK4c/HMjUXuTAEAAOTAZAoAACAHJlMAAAA5MJkCAADIYcEG0EsHj7v2aLwCpKRKFL5sL/mQZy0jENfK8KF0ceTKxjQ8OlkpY3HRzn0Ldx686j23uPYbr/uJZJ/fXfF117684tONoxnV6b45vNG1z2k74tqXth1L+tw+5hOdyzMCuoXoSwnbXvh3focXJl303TE/Jg7U/cKnnW/zCwlnKkTPsZGOo5kw3ptu2zq0PNpytPWBTtP5zXXFER/GLWcEkldWfAHD+NrzcC1KFksKtanLtt57aHWy7dwNfsyvbPePm7XQcagu3GvRTLCML140OqNgeFSk89xl/nWSpD0V/+WmWkeyi6Lar4rrw4b0z6AUvb7W5jtlFeBstJhZxIt7S1K102/siP7eHhpNVzGud0S/p9qZ+xIL7wIAAIAcmEwBAADkwGQKAAAghwWbmao9vN21n7YsLZz41QOXTHmM8VPITJUyim3Gi5hOR9feM7eg41zz7b3nJtves8YX3Pz3YR/s6Sn4oq2S9JSO7a5djj7X31HrTPosKfjiefWsj+yj49w97vscrafHlXwQ4eahC/0hv50WEE2E/GOk0JWeW2PQ52iqvemT3nLYF4RcEWWmCl1pdjBejHqhuuH4pcm2uFjmiijL9KmjV2ccaerX/9i+NHfXea5/nEY0eIfjqo+SbIx/k+dRqKbvH6v733t3h89dHjiR5od6t/m/I8fa0j/vxajWZ1ykM+tPUVwetD7i/4bFx5Sk9qhIZ/sR/xzjfJQkLd7qM1L3bfcLzT/14i1Jn2RN8DNY95d3AQAAQA5MpgAAAHJgMgUAAJADkykAAIAcFmwAPXa4mgb6SlG6rR783LNrAOoAAAAefUlEQVSRVXmshULGouvVqIBkte7bjZDOeTsOTV3ocyFZ/s72ZFv50/53+JxOX3CzkPHviJ1R+HJL1YfW++tpaLpaPOHaPYU0jdlT8K9VOQoGt1v6Wm4o+YD8/3rzj7t2p25L+pyOIp2WtaJ9pJ6xAnz/Yf9+WhH93Ion/+WN+Wpwo29f1fFwss+do+td+5L2va79js3PTvqs0uYpH7djd0ah4misLin7LwVsbD+c9OldNzDl42BqSYhaUmnQX58Wtfvryp5b1iZ9Bq7x7fLGwWSfRiG69rT51/vEXUuTPoVOf2FsDPtpw+jy9P3fd1laVNQdo5pOPXZu8l+IKO3z+5QuzSgYGx83/tbQLOLOFAAAQA5MpgAAAHJgMgUAAJADmamm/mpanLBSmrqY5qN79yTbblFa1G6yeGFGKV08tC1aYLm77Au2SdKxDp85yVqfcqHIKmD5rDWPce3jv/AE1772j76T9Pm/K/1xzivHn9EfV2tZr//UYyLLr+16lmt3fjYjIzULQr117uqxj9+abNt8cOXUxw1nsLreHNP3gG9/+thVyT53HF3n2s9dfa9rj97bd9KPu+ye9Pq2+QV+8eMdJ5a4dlaB4cY3liTbMH2loYwskPnre6Xof+/rvzKc9Km+pd+1L+w9mOxzYMTnkvrH/GrI9QNp5mi46DOplSG/T9bftHP7fGbqRNUXIV7bmebsxlb557z3T8937UNPTnPN8dlmLRo9W7gzBQAAkAOTKQAAgByYTAEAAOSwIDJTVkqfZqj5z6C/9j+XJfv875/4nGvvHvfZgO6sFR5b5GO6t6fz12cuujdjzx8qWPo58JFdvr4Ryx5PbdENPiN15w3pPs+Sz1nZ4/yYOHC1rzslSf2P8uOoe/WJZJ+1vT4fEKL6ZA8dWJb0Oe+6FgsZZ9V/moG6Uskhh9NsRmzf+89Ltm2422cm4jMLI+lC0wtV3/W3uva916f7tGmHa39NPa69Uf4Y09Hxue8m23Z8yV8rQ83nQm/P+Pf3Kt1y0o+NH7KM/GCtN8rRFnzbdqb1vuwZ/rXanvlo/n1XiX668kf0OlmtKo/tnMYxKuf6a8h4I/07Xl3kfy+hcObuD3FnCgAAIAcmUwAAADkwmQIAAMiByRQAAEAOCyKAHofNs1z6ljQS9577X+LaVZ/5zCx616E01DnZ6o+mweJXbfx1147WPVbH/nTOu/ZOQp+nW7j9B6694vZ0n3gB38zjtPj5edo97XP64UFnqTjdNB6n+9NpQdFWUfjpvCcXikK7L4rYGM36YsspHLfLL8wdxv2itqGarrre6nXJ+jKPtfkv3UznSwv4oaFVGV9aqvjX6u4dfmHji088dDpPaU4I/b5I8pY9G5N9Cj3+91SonnyB5JnCnSkAAIAcmEwBAADkwGQKAAAgB2PBUQAAgFPHnSkAAIAcmEwBAADkwGQKAAAgByZTAAAAOTCZAgAAyIHJFAAAQA5MpgAAAHJgMgUAAJADkykAAIAcmEwBAADkwGQKOEuZ2XYze/qZPg8AWOiYTLXQ/IM1YmaDZtZvZreY2avNjN8d/n9mdk1zbAyY2VEz+7aZXXWmzwvzB9ci5MUYOn34BU7PC0IIPZI2SPoLSX8s6aNZO5pZcTZPDGeemS2S9CVJfy1piaS1kt4iaexMntd0mVnpTJ8Dpo1rEfJiDJ0GTKZOQghhIITwBUk/L+mXzexRZvaPZvZBM/sPMxuS9FQzq5jZu8xsp5kdMLMPmVmHJJnZMjP7UvNfBUfN7FuP/KvAzP7YzPY0/9Wwxcx+8gw+XUzfhZIUQrghhFAPIYyEEL4aQrjbzF5hZjc3x8MxM9tmZs95pKOZ9ZrZR81sX/O1f/sjFzAzO8/Mvm5mR8zssJn9i5n1ZZ2AmV3cPPZLm+01ZvavZnaouf13J+37ZjP7jJl93MyOS3rF6fzlYOZxLUJejKGZxWTqFIQQvitpt6SnNDddJ+nPJfVIulnS/9XEH9jHSDpfE3cq3tTc93XNvsslrZT0BknBzC6S9NuSrmr+q+FZkrbPwtNBfg9IqpvZP5nZc8xscfTzqyVtkbRM0jslfdTMrPmzf5JU08Q4eaykZ0p6VfNnJukdktZIukTSOklvjh/czK6U9FVJvxNC+GTzYvZFSXdpYuz9pKTfN7NnTer2IkmfkdQn6V9O/anjTOJahLwYQzODydSp26uJj3Qk6fMhhG+HEBqa+Gjn1yT9QQjhaAhhUNL/kfTS5r5VSaslbQghVEMI3wohBEl1SRVJl5pZOYSwPYTw0Kw+I5ySEMJxSddICpL+XtIhM/uCma1s7rIjhPD3IYS6JiZPqyWtbP78OZJ+P4QwFEI4KOk9ao6VEMLWEMJ/hRDGQgiHJL1b0o9HD/8USV+Q9MshhC81t10laXkI4a0hhPEQwsPN83rppH63hhA+F0JohBBGZvY3glnGtQh5MYZyYjJ16tZKOtr8712Tti+X1Cnp9uatz35JX2lul6S/lLRV0lfN7GEz+xNp4g+npN/XxJ2Hg2b2STNbc/qfBmZCCGFzCOEVIYRzJD1KE3eT3tv88f5J+w03/7NbE5mFsqR9k8bKhyWtkCQzW9EcB3uaH8d9XBN3tyZ7taRbQgjfmLRtg6Q1jxyzedw3aOJfjo+YPGZxduNahLwYQzkxmToFNvEtrbWauAUqTdyReMRhSSOSLgsh9DX/1xtC6JakEMJgCOF1IYRNkl4g6bWPfJYcQvhECOEaTfwxDJq4vYqzTAjhfkn/qIlJ1VR2aeJffssmjZVFIYTLmj9/hybGwRUhhEWSflETH/1N9mpJ683sPdFxt006Zl8IoSeE8NzJp3lqzw5zCdci5MUYmhlMpk6CmS0ys+dL+qSkj4cQ7on3ad4a/XtJ7zGzR+4wrH0kr2Jmzzez85uZmeOauB1aN7OLzOxpZlaRNKqJAVyfnWeGPJrh79eZ2TnN9jpJvyDpO1P1CyHs00TW6a+aY6tgE6HzRz7K65F0QlK/ma2V9PqMwwxKeraka83sL5rbvivpeDMA2mFmxWa4lFIN8wTXIuTFGJpZTKam54tmNqiJf/G/URPZlV+ZYv8/1sStz+80P575mqSLmj+7oNk+IelWSX8bQvimJj5f/gtN/EtgvyY+6nnDjD8TnA6DmgiZ39b8Bsx3JN2riXBmKy+X1CbpPknHNBEKX9382VskXSlpQNK/S/ps1gFCCP2SniHpOWb2tmY26wWaCIxu08SY+oik3lN5cphTuBYhL8bQaWATWTEAAACcCu5MAQAA5MBkCgAAIAcmUwAAADkwmQIAAMhhVhc4fUbhZ09P2t2i0junEKof/umrk239m/waj2vedctJH3cm7H7Dk5Jty+6uuXb7l747W6fT0n81Ph3XQppRp20cYU45nePobB9DxWVLXbt++MiMHLf2tMe5dunrt8/Icc8UrkVTYxxNz3TGEXemAAAAcmAyBQAAkAOTKQAAgBxmNTN1ppRWrUy2Lf7smGtf0PWtZJ/hRptr33iZ/xzYihkflzeij1YLfp++vqGkSwi+zyXLDrj2c/r+I+lTvc7nuQ6/rce173xsemoA5r4HPpyu+vOHT/mKa3cV9rp22dKVOt65+Zmu/dPn3uXav7I4zVnurflsy5cHr3Dtz3/0xxVb+f4zkyXF1BhHs4s7UwAAADkwmQIAAMiByRQAAEAOTKYAAABymB8B9BZFOrd/YFmy7bUrP+nanzj0xGSfkXrZtZ/3qHtd+7sH1yd9VnSdcO2H/9+5/lQfP5z0GR71Qfd1ncf8MUaWJ31qwQfQX7702679lT/4naTPqvec3QE/YD568H1PcO0Hnv+BZJ8vD/svmBSt4dpLC+kXW959+Y2u3W5V1364uijps7O6xLWf2LXVtV/7R99P+rxg5++5dsfn5k4B4YWEcXRmcWcKAAAgByZTAAAAOTCZAgAAyGF+ZKZiBZ8nesbGLckunz92pWs/pmdXss+dg+tcu6M47tovXHdP0mfvWJ9rH3tyh2v/zDl3Jn0G6+2uHX+OXWukc96Oov/c+usnLnXt3/31zyZ9bnzPqmQbgDPrpdf6LOPmajXZZ7hRce1CdI0YrPvrTNY+ow2fAS1amjVdVBx17f21Xn9u42nmc+Mf3+/aBz6X7IJZwDg6s7gzBQAAkAOTKQAAgByYTAEAAOQwLzNTD3zQ56GeUErrK52o+c+OBzI+Kx5r+F/P4fFu117Rdjzpc27HIdc+Z42vGRV//iyli0vGuas4qyVJy8qDrj1c989n5/jSpM+RV/laWks/cmuyD4DZ9bzeNEcZay/4/EtB/jrSsGn8uzjapRHSPvFxY/vrvcm2j67/hms/X49L9sHpxzg6s7gzBQAAkAOTKQAAgByYTAEAAOTAZAoAACCH+RFAj4p0PudKX0zzwvb9SZc7Tmxw7WPVztYPI1+cbM/Y4mSfJ/U8OOUx4kUiJenrx33BzTiknhV0j0PrGyqHXXtVaSDp8+CvrHDtIx+Z8lQBzIInt/t/0347WvhckupRyDe+RoxHC59LaTA43mc0pI8T79Nf99fFnoIvxihJZfN9ipdc4Nr1zVNfEzEzGEdnFnemAAAAcmAyBQAAkAOTKQAAgBzmR2aq4fNDD13l29/6t2clXV5z0U2uff/I6mSfcsEfp1KsufZI3S/4KEl3D6937Ud17HbtmwYvTvrEx1lV8RmpaiN9mZa3+aKdl1f847xm83VJn97nbk22AZg9Vqm03Kca0vd7vPh5vGBtnKGUpOHgrytxIcViRmHFtug48XH7ikMZZ+zte+py114xx7MuZyPG0dzDnSkAAIAcmEwBAADkwGQKAAAgh/mRmWphzYvvS7Zd/5WrXftjl1yf7POOfc927SVl/zlvI1jSJ/5s+O7hda5dz5i/rqn4mlAry74df64tSRe0+dpZL7v+91x7w5tYxBiYawrnb8zYeptrZdX6GW343Ep8HSkrzbpk1bSbrJFxLRqKrjVLiyemcUx/nJGVUz4sZgDjaO7hzhQAAEAOTKYAAAByYDIFAACQA5MpAACAHOZnAD1a+Dgu6ilJ3c9+2LV//7JfTfZ525d8KP0TR5/g2otKWYs1+sdqFHxIvdvGWvaJvWbxlmTbiy/5SdfecJzA+Zxg6ZcSEsEvmG2l9G0Y6tGYiPuU08VFQ3W89WPHpvFeORVxUcEwHp1b9HwWisOPX9Jyn6xA72hUODG+ZlQzwsY9xRHXjhe5Ha9nBJSjx1lXPuLav3/fS5M+37vyRn/cc9PrImYW42ju4c4UAABADkymAAAAcmAyBQAAkMP8yEzFOZUo95GZSan5RYu192Cyz4aSLyxWMJ/zyMo6lQv+uNXo8+RKIc21tBemfpyKpQsq148fT7ZhDjiVLJBl/Jsm1NJtk398Cvmo3W94UrLt/a/8sGu/87zLT/q4WcJYmg2ENHB+633WlAaSbf31zin7xBkVSVoiXyixaK3HZnztWVUcdu2hW5elna6MjlFKF77FzGIczT3cmQIAAMiByRQAAEAOTKYAAAByYDIFAACQw/wIoLcI/YZG68BcY3i45T6dRR/67S2lfY7Vulw7DqnHYfMsXQUf3h1unEIxxqzikQu0UOIZF78W0etwKmHyg69Jw+T9l/vQ+rue9knX3l/zhfMk6fvDm1z78BcvdO1lL3jgpM9Nkgrt7a794Nse69rnvX5hFpkdX9v6/X/H6PpkW0/BFzCMg8KjjTQ4PNjocO26/DgcbvjCqlIaHI771LpPPnyMmcc4mnu4MwUAAJADkykAAIAcmEwBAADkMD8yU62E1sW/sooMDkRZq+6i/7x5uJEuNhvrifoULD2XsUa8+KTPvuyrn0JmCrOjRR7qR26bfIjHXpZse+ili1x70+N3ufY3L/qrpM/Hj/u801f7/XF3DS1O+jxnxQ9c+8Yr/sG1f0vXZJxxa3t/w1fgO+/Knad0nPlmyfLWxXbf9bGXJNve/qp/du12+czMoHxGTZKGoutTXGyxGjIu/9FQ7Y+OUetqfS3t6qRg6+nGOJp7uDMFAACQA5MpAACAHJhMAQAA5DA/M1PTybFMw98d9XmRC9v3u/bu8SVJnzgjVYnqSrVbmn+qBr8Yclxn6v5qxqKQODkF/zuOF8OW0tpIjdHRZJ/ENMZWceUK197yrrWu/a/XfCjps6fe69rfPH6Ja//R3qclfbqLftwsb/MLlH7j4QuSPsPLfJbhude/3rU3Kq0HVdqwzrW3vXxdss/3f+O9rv0zz/tl1x5/2uPS43799mTbfFMuts6KrPvy0WTb6K/6XOXyks/MHKr1JH3iLEs1GqpZ+c24Ll5XlN/s3BO9jyTVo0xqb8c03jfIhXE093BnCgAAIAcmUwAAADkwmQIAAMiByRQAAEAO8yOAPkOB89j6il8YNl7QMQ7ZSdLRaKHjDW2HXfvh8eVJn/YonLen6oPtcVFPSSqtWunatf0H/A6WMU8O6fnOW9GYsIJvZ9VxnVbgPDL0kqtde99PpV8w+PJT/sa17xg9x7U/cDANk4/U/Wu+sdOPxSu6dyd9DlZ9oc/9Y7798ku/m/S57dhG177uBTe59rOuuyfps7++1bU/uPMnkn1evP6Jrl3s3uPa7f0+YC9JtWTL/FMstA4O20g6hi5o8+/vPbW+lseJF7EtqjFlW0qvNcujoHPIuKw0ogqNnWV//gvoqjNrGEdzD3emAAAAcmAyBQAAkAOTKQAAgBzmR2aqVUYqLtgoJUUb9/7hk5Jdugqfce3NI2tce2U5XWxyLCpwFn/efKKeLiTZU+537T3jfkHaa7vvT/q8791Pde3zrosyUxlFKReUaEyE2skncna+yY+J337pF5N9ntL5Ptf+8uDlyT7vPfiTrh3noa5e9HDLc4kL5zUyQgdxhq/W8OP+zgGf1ZKk9V3HpnzcP9n6M8m2yjO3R1vS/NZDf+kzUx/56Q+79hf7H5P0ue+Vl055LvOBtd5FdmI42XZ+2b+2d44uSvaJZWVZJitYet0cbfhxNhgt9l4aSY8z0PBZw1KU51ngV6LTgnE093BnCgAAIAcmUwAAADkwmQIAAMhhfmSmWskqKhQpPSVdFHJoGnWlYstKfnHZ/nqna/cW0w+L4+NWkrpTPkMlST93yR2uffsCnhfXn3plsm3nM/1rVzzfvy4dlbQGy6NX7HXtq9q/5dpbhlclfW46eqFrn9t1JNmnr+SzC+d3+HOpZ7x2+8Z9/Zd4Ae2sxUVHo9ouHUW/yHY1I2d1eKzbtY+O+/H6v85Lc2LFh3z+YUMpzQ7+x5D/PVx/yOfPVlbSPve/ujvZNt9MqwJeIX2d2m3qLGbWtakRfLImGWcZ18ViXOsnCufU/PBonovvMzju33t+KW3MBMbR3LNw/wIDAADMACZTAAAAOTCZAgAAyIHJFAAAQA4LJIDeOq530bKDyba4MGJnwYeWBzMKcPZGYeOewphrD2Qk77qL/nGGGz5q18iY8x6vdURbxpJ9EnHx0rO0sOeuP/Nh5iufe1+yz6MqPuQdF55Lf39SV8n/Dg9ECwVnhb7XdAy4dq2Rvla7Rv0XCLYGv9h1exQUnziOf62WtPlxlXUui8t+n0rBH3d5W/o4S8tDrh0H3R8cS0P3cWj1now4bDyGl0Wvx8Z2vwD4QnFitJJsq0cB3tCZXlcK0TWgGvz4aLf0tY0XO69PI7Ucj5n4qI2MvxjxSNy+e5lrX6gdrR8YJ4VxNPdwZwoAACAHJlMAAAA5MJkCAADIYX5mpiyqEDaNzNST+x5Kto1Gi8suKw+69s6xpUmfuOhZPcpdxQshS2meJ85mxcUYJen8Tr+w8YOF6FzO0jzUdGz88IOuved75yf7fP/J0e/sYp/ZeczaPUmfDR2+0OSlnb6IZ1chzaXFr03Z0gWVr+r2r8XV7btcu5qxbGl7tHhob5R367S0hF3ZMhb0nmRn7USybVeU4etv+HZcuFZKs4SHauliqb1Fn9/aM+aLkB6rdSV91n052vDryS5nvUo5HR/FOJOytHXx0vg1iLMvktQe5VbaorGZdV2Jr0V7o2zh+AVp0eH1peh8a/wb/XRjHM09Z9fZAgAAzDFMpgAAAHJgMgUAAJDDPM1MxQsxts4PbWg7lGzbMrbatdvNZ5nGsoplRIoZ9YBi8cKRDbVYWFLSqpKvb1RacZFr1/b7TNW8UvC/n47bHkx22fCfA8m2yQY603pfN192lWsfu9h/hj+4Ic02ja72YytUMsZa3K0QZfga6XFLR3wOoTTk96mk63Kr0u+P294fLaB9NF3cuXjC58AKg2mWIRbao7xWnFHMstfXcdvSn2Y+OsJ3Wx/nLDf8reXJtk0P/YZvt2fU+onEdcYKluZCO6OMX5rvS8dqXEPskrZozBxOM3QXXP+brr08LfuGGcY4mnu4MwUAAJADkykAAIAcmEwBAADkwGQKAAAgh/kZQD8FX+2/PNm2qcOH0oejAoaNkAZv4zBeISpe1h0tJJt1nEK0cGxcWE2S+qKiiI2VS/wO8ziAXj/gw8zFvt5kn9Kmja4dCq1D0oWD/a69dOtu117WlYbWw1ga6o5ZKSqEFxeRLaaF8pJFSqNjhEpaKK/R5vepd/p9xhelfWqr/Jge7/HFNTPq8Smq4Ze5aGmt0/++y4N+fBaradB10bb0vTHfrP2LW1ruM/bcq5JthehbDPH7Py4OLKWL1g7Lv9ZZBWYHG37cxcVh246lj7P+ra2fE2YW42ju4c4UAABADkymAAAAcmAyBQAAkMO8zExZlI8JGXUzreSf+vK2wWSfevT5cly8LKuYZpx/ai+2LpwWixeWzFp8Ms5iDa/r8ce466Qf9qxV788o0Jm1rYVCj/8dWiUqTllLswHq831CR7oAcaNt6rdZKKXjKM54Wa118ddQ9MexKJvV1p/muzq3+8xEXIAzlDPyXPHzyTq3+DlF+xQGo8eVVN+6LT3OPGPldHyEqn9d6pV0PPwg2id+/5cLaeHEeGHueEHqzmK6cHdP8Lm1u8f9cceXtB6H8bU1ZL1vkAvjaO7hzhQAAEAOTKYAAAByYDIFAACQw7zMTIVGWsMmVujucu2ypZmpYlTvqRgtAllU6899k2MoPbc4exUvPpmVmYoXUK51Mi/OqzEYjYF0SKT2n5ZTSdZGPpU+0zlG6yXAT48z9bhnXFaAM1IeSn875eha0xbXs8tYUD2ueRdfR+L6QZI0qI7oGP64GSWFEtO5/iInxtGcw19gAACAHJhMAQAA5MBkCgAAIAcmUwAAADnMywD6dIRxH5prZMR1WwXvCpYG5OJtceHPSrxKrNKFjOOQeiMrtB71KY20DiQCmPsK1fS93BYFeOMvrWSFgNuimP9o8KtWx+HjLF1RUrhQO5WvRuBMYBzNLu5MAQAA5MBkCgAAIAcmUwAAADks2MxUY9gvtDpcTxeOXFbyVRvjnNIpPe4pHCOrkFqsrb/1gsrTWQAawJlVHErfy/FVoxxlUOK2JLVH2+IsZrzIrZQusl6Ooi2FsfmXdZmvGEeziztTAAAAOTCZAgAAyIHJFAAAQA5MpgAAAHJYsAH0WH+tM9l2Ycd+1x4PJ//rSsJ6GWHyVqtwFzLmvFX5PuX9A66dVUbtbFuFG1iICsOtv0wSF0osZ7zjK9E+ccHgrGKLxShMXI5+3jYgnCUYR7OLO1MAAAA5MJkCAADIgckUAABADmSmmq5dtKXlPv11n6vKWug4Fi90HC+eLKWZqWqUzYqPIUmD9Q7XtsGhludC0U5g7rN9B5Ntgy3ymnGGUpKGoz6Hqz2uvbZ8NOkTL2JbNH/N6N3eelFbrjNzA+NodnFnCgAAIAcmUwAAADkwmQIAAMhhfmamTuHD1buH1yXbru3xOar9tV7XPqftWNJnY/mQay8t+ixTf/FE0if+fDm2v9qXbOssjE3ZJwt1poAzK9RbZ0XqR9IMyvdGNrr2s7q2unaadJFWl7pd+4q2+1x7Z20k6bO26AsAlaO85qLb9yZ94qVxp/MckQ/jaO7hzhQAAEAOTKYAAAByYDIFAACQA5MpAACAHOZpAP3kg9a3Hj432bax/bBr7xv3QfAtx1cmfb5Qu8K1l7b7APpoLQ2bx4sf16Iinpu6/XlI0rKyD7LXDx9J9gEwx5zCtUmS3vmZF7v2v16zw7V3/ufGpM+KO8Zde9vP+hCwlTO+qDPk/yR07fDXojU7bml1qpgNjKM5hztTAAAAOTCZAgAAyIHJFAAAQA4WTvGzVwAAAHBnCgAAIBcmUwAAADkwmQIAAMiByRQAAEAOTKYAAAByYDIFAACQA5MpAACAHJhMAQAA5MBkCgAAIAcmUwAAADkwmQIAAMiByRQAAEAOTKYAAAByYDIFAACQA5MpAACAHJhMAQAA5MBkCgAAIAcmUwAAADkwmQIAAMiByRQAAEAOTKYAAAByYDIFAACQA5MpAACAHP4/axtL3o9mw7AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 8 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "idxs = np.random.randint(100, size=8)\n",
    "f, a = plt.subplots(2, 4, figsize=(10, 5))\n",
    "for i in range(8):\n",
    "    X = train_data.train_data[idxs[i]]\n",
    "    Y = train_data.train_labels[idxs[i]]\n",
    "    r, c = i // 4, i % 4\n",
    "    a[r][c].set_title(decode_label(Y))\n",
    "    a[r][c].axis('off')\n",
    "    a[r][c].imshow(X.numpy())\n",
    "plt.draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a model\n",
    "\n",
    "As we are more focussed on evaluating how the different optimization methods perform, we'll be constructing a very simple feedforward network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FashionModel(\n",
      "  (fc1): Linear(in_features=784, out_features=64, bias=True)\n",
      "  (fc2): Linear(in_features=64, out_features=32, bias=True)\n",
      "  (fc3): Linear(in_features=32, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "class FashionModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(FashionModel, self).__init__()\n",
    "        self.fc1 = nn.Linear(784, 64)\n",
    "        self.fc2 = nn.Linear(64, 32)\n",
    "        self.fc3 = nn.Linear(32, 10)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = F.log_softmax(self.fc3(x))\n",
    "        return x\n",
    "print(FashionModel())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Simple Optimizer in PyTorch\n",
    "\n",
    "The most simple optimization method is to update the network parameters by adding the negative of the gradient scaled by a fixed learning rate $\\eta$.\n",
    "\n",
    "$$ \\textbf{W'} \\leftarrow \\textbf{W} - \\eta \\nabla L(\\textbf{W}) $$\n",
    "\n",
    "PyTorch provides us a very simple and expressive API for creating custom optimizers. An optimizer in PyTorch subclasses `torch.optim.Optimizer` and is required to specify two methods.\n",
    "\n",
    "`__init__`: Must call the superclass `__init__` and provide a list of network parameters, `params`, to optimize and a dictionary of default values provided to each parameter group, `defaults`. \n",
    "\n",
    "`step`: Performs an update on the network parameters. \n",
    "\n",
    "The meat of your optimizer logic lies in the `step` method. In this method you should update your model parameters with the help of some useful internal datastructures. Let's define these to make the following code more clear.\n",
    "\n",
    "`self.param_groups`: When you initialize an optimizer object, you are required to provide the list of parameter objects to be optimized. In the case of `FashionModel`, there are 6 parameters -- each `Linear` layer has a weight matrix parameter and a bias vector. All of these 6 parameters are considered within a single `param_group`. This `group` will be a dictionary with an entry `params` that contains an iterable of all 6 parameters, as well as entries for all `defaults`. These `defaults` are generally useful for storing small values like hyperparameters that are standard across all parameter groups. There are more advanced cases where it can come in handy to have different values for certain entities depending on the `param_group`. \n",
    "\n",
    "`self.state`: This maintains state for a given parameter. Essentially it maps a parameter to a dictionary of data that you want to keep track of. This is useful in cases where you want to keep state on a per-parameter basis.\n",
    "\n",
    "**IMPORTANT**: Unlike most other use cases in PyTorch, operations on parameters and state data should be done inplace. This ensures that the updated parameters are not updated copies of the original. In the following sample implementations, you may see some unfamiliar operations. Functions like `torch.add_` and `torch.mul_` are just inplace analogues of standard PyTorch functions. See http://pytorch.org/docs/master/torch.html for further details.\n",
    "\n",
    "\n",
    "## Let's write our *Trainer*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_size = train_data.train_data.shape[0]\n",
    "val_size, train_size = int(0.20 * train_size), int(0.80 * train_size) # 80 / 20 train-val split\n",
    "test_size = test_data.test_data.shape[0]\n",
    "batch_size = 100\n",
    "\n",
    "# Add dataset to dataloader that handles batching\n",
    "train_loader = torch.utils.data.DataLoader(train_data, \n",
    "                                           batch_size=batch_size,\n",
    "                                           sampler=torch.utils.data.sampler.SubsetRandomSampler(np.arange(val_size, val_size+train_size)))\n",
    "val_loader = torch.utils.data.DataLoader(train_data, \n",
    "                                           batch_size=batch_size, \n",
    "                                           sampler=torch.utils.data.sampler.SubsetRandomSampler(np.arange(0, val_size)))\n",
    "test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, shuffle=False)\n",
    "\n",
    "# Setup metric class\n",
    "Metric = namedtuple('Metric', ['loss', 'train_error', 'val_error'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def inference(model, loader, n_members):\n",
    "    correct = 0\n",
    "    for data, label in loader:\n",
    "        X = Variable(data.view(-1, 784))\n",
    "        Y = Variable(label)\n",
    "        out = model(X)\n",
    "        pred = out.data.max(1, keepdim=True)[1]\n",
    "        predicted = pred.eq(Y.data.view_as(pred))\n",
    "        correct += predicted.sum()\n",
    "    return correct.numpy() / n_members\n",
    "\n",
    "class Trainer():\n",
    "    \"\"\" \n",
    "    A simple training cradle\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, model, optimizer, load_path=None):\n",
    "        self.model = model\n",
    "        if load_path is not None:\n",
    "            self.model = torch.load(load_path)\n",
    "        self.optimizer = optimizer\n",
    "            \n",
    "    def save_model(self, path):\n",
    "        torch.save(self.model.state_dict(), path)\n",
    "\n",
    "    def run(self, epochs):\n",
    "        print(\"Start Training...\")\n",
    "        self.metrics = []\n",
    "        for e in range(n_epochs):\n",
    "            epoch_loss = 0\n",
    "            correct = 0\n",
    "            for batch_idx, (data, label) in enumerate(train_loader):\n",
    "                self.optimizer.zero_grad()\n",
    "                X = Variable(data.view(-1, 784))\n",
    "                Y = Variable(label)\n",
    "                out = self.model(X)\n",
    "                pred = out.data.max(1, keepdim=True)[1]\n",
    "                predicted = pred.eq(Y.data.view_as(pred))\n",
    "                correct += predicted.sum()\n",
    "                loss = F.nll_loss(out, Y)\n",
    "                loss.backward()\n",
    "                self.optimizer.step()\n",
    "                epoch_loss += loss.data[0]\n",
    "            total_loss = epoch_loss.numpy()/train_size\n",
    "            train_error = 1.0 - correct.numpy()/train_size\n",
    "            val_error = 1.0 - inference(self.model, val_loader, val_size)\n",
    "            print(\"epoch: {0}, loss: {1:.8f}\".format(e+1, total_loss))\n",
    "            self.metrics.append(Metric(loss=total_loss, \n",
    "                                  train_error=train_error,\n",
    "                                  val_error=val_error))\n",
    "         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SGD OPTIMIZER\n",
      "Start Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/tf3/lib/python3.6/site-packages/ipykernel_launcher.py:11: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  # This is added back by InteractiveShellApp.init_path()\n",
      "/anaconda3/envs/tf3/lib/python3.6/site-packages/ipykernel_launcher.py:43: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1, loss: 0.12873552\n",
      "epoch: 2, loss: 0.02345240\n",
      "epoch: 3, loss: 0.02074262\n",
      "epoch: 4, loss: 0.01942439\n",
      "epoch: 5, loss: 0.01835329\n",
      "epoch: 6, loss: 0.01766403\n",
      "epoch: 7, loss: 0.01695254\n",
      "epoch: 8, loss: 0.01645464\n",
      "\n",
      "ADAM OPTIMIZER\n",
      "Start Training...\n",
      "epoch: 1, loss: 0.26999099\n",
      "epoch: 2, loss: 0.01946122\n",
      "epoch: 3, loss: 0.01242979\n",
      "epoch: 4, loss: 0.00995855\n",
      "epoch: 5, loss: 0.00878618\n",
      "epoch: 6, loss: 0.00801902\n",
      "epoch: 7, loss: 0.00758505\n",
      "epoch: 8, loss: 0.00729035\n",
      "\n",
      "RMSPROP OPTIMIZER\n",
      "Start Training...\n",
      "epoch: 1, loss: 0.22505141\n",
      "epoch: 2, loss: 0.01458840\n",
      "epoch: 3, loss: 0.01202629\n",
      "epoch: 4, loss: 0.01174371\n",
      "epoch: 5, loss: 0.01005420\n",
      "epoch: 6, loss: 0.00986540\n",
      "epoch: 7, loss: 0.00877988\n",
      "epoch: 8, loss: 0.00779939\n",
      "\n",
      "Test accuracy of model optimizer with SGD: 39.85\n",
      "Test accuracy of model optimizer with Adam: 71.88\n",
      "Test accuracy of model optimizer with RMSProp: 61.99\n"
     ]
    }
   ],
   "source": [
    "### LET'S TRAIN ###\n",
    "\n",
    "# A function to apply \"normal\" distribution on the parameters\n",
    "def init_randn(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        m.weight.data.normal_(0,1)\n",
    "\n",
    "# We first initialize a Fashion Object and initialize the parameters \"normally\".\n",
    "normalmodel = FashionModel()\n",
    "normalmodel.apply(init_randn)\n",
    "\n",
    "n_epochs = 8\n",
    "\n",
    "print(\"SGD OPTIMIZER\")\n",
    "SGDOptimizer = torch.optim.SGD(normalmodel.parameters(), lr=0.01)\n",
    "sgd_trainer = Trainer(normalmodel, SGDOptimizer)\n",
    "sgd_trainer.run(n_epochs)\n",
    "sgd_trainer.save_model('./sgd_model.pt')\n",
    "print('')\n",
    "\n",
    "\n",
    "print(\"ADAM OPTIMIZER\")\n",
    "normalmodel = FashionModel()\n",
    "normalmodel.apply(init_randn)\n",
    "AdamOptimizer = torch.optim.Adam(normalmodel.parameters(), lr=0.01)\n",
    "adam_trainer = Trainer(normalmodel, AdamOptimizer)\n",
    "adam_trainer.run(n_epochs)\n",
    "adam_trainer.save_model('./adam_model.pt')\n",
    "print('')\n",
    "\n",
    "\n",
    "print(\"RMSPROP OPTIMIZER\")\n",
    "normalmodel = FashionModel()\n",
    "normalmodel.apply(init_randn)\n",
    "RMSPropOptimizer = torch.optim.RMSprop(normalmodel.parameters(), lr=0.01)\n",
    "rms_trainer = Trainer(normalmodel, RMSPropOptimizer)\n",
    "rms_trainer.run(n_epochs)\n",
    "rms_trainer.save_model('./rmsprop_model.pt')\n",
    "print('')\n",
    "\n",
    "\n",
    "### TEST ###\n",
    "model = FashionModel()\n",
    "model.load_state_dict(torch.load('./sgd_model.pt'))\n",
    "test_acc = inference(model, test_loader, test_size)\n",
    "print(\"Test accuracy of model optimizer with SGD: {0:.2f}\".format(test_acc * 100))\n",
    "\n",
    "model = FashionModel()\n",
    "model.load_state_dict(torch.load('./adam_model.pt'))\n",
    "test_acc = inference(model, test_loader, test_size)\n",
    "print(\"Test accuracy of model optimizer with Adam: {0:.2f}\".format(test_acc * 100))\n",
    "\n",
    "model = FashionModel()\n",
    "model.load_state_dict(torch.load('./rmsprop_model.pt'))\n",
    "test_acc = inference(model, test_loader, test_size)\n",
    "print(\"Test accuracy of model optimizer with RMSProp: {0:.2f}\".format(test_acc * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHAVJREFUeJzt3X+QVfWd5vH3Q3fTIgKitkZp2DZqjGgsJS0GOTipmBjcSSSpNRs0mTFTpsxWrVv5tZkxU5tfOn8kOztxZnasrVgxKU006DibWjYx4yZrJoOIQKNRgorbEpUGUQiKgAjd8Nk/zuni2tzuvt19+55773leVaf63nO+595PU9TzPf25556jiMDMzIphSt4FmJlZ7Tj0zcwKxKFvZlYgDn0zswJx6JuZFYhD38ysQBz61tQktUjaJ2leNceaNSr5PH2rJ5L2lTw9HjgIHM6efy4i7ql9VRMn6a+Azoj4TN61WLG15l2AWamIOGHwsaQXgM9GxK+GGy+pNSIGalGbWTNwe8caiqS/knSfpJ9I2gt8WtIiSY9Jel3Sy5L+XlJbNr5VUkjqyp7/ONv+C0l7Ja2RdOZYx2bbr5L0nKQ9kv67pNWSPjOO3+l8Sb/J6t8o6Y9Ltn1E0jPZ+/dJ+mK2/lRJD2b77Jb0r+P9N7VicehbI/o4cC8wC7gPGAA+D5wCLAaWAp8bYf/rgK8BJwEvAbeOdaykU4H7ga9k7/t7YOFYfxFJU4GfAT8HOoAvAvdJOjsb8kPghoiYAVwI/CZb/xVgS7bPO7IazUbl0LdG9EhE/O+IOBIRByJifUSsjYiBiNgC3AH80Qj7PxARPRHRD9wDXDSOsR8BfhsR/yvbdhuwaxy/y2JgKvDXEdGftbJ+ASzPtvcD8yXNiIjdEfF4yfozgHkRcSgifnPMK5uV4dC3RrS19Imkd0v6uaQdkt4AbiE9+h7OjpLHbwInDDdwhLFnlNYR6RkRfRXUPtQZwEvx9jMqXgTmZI8/DlwNvCTpXyRdmq3/djbu/0p6XtJXxvHeVkAOfWtEQ085+x7wO+DsiJgJfB3QJNfwMtA5+ESSOBrUY7EdmJvtP2gesA0g+wvmauBU0jbQimz9GxHxxYjoAj4G/IWkkf66MQMc+tYcZgB7gP2SzmPkfn61/AxYIOmjklpJP1PoGGWfFknHlSztwKOkn0l8WVKbpA8A/xa4X9I0SddJmpm1kPaSnb6ave9Z2WSxJ1t/uPzbmh3l0Ldm8GXgetJQ/B7ph7uTKiJeAT4JfBf4A3AW8ATp9wqG82ngQMmyOSIOAh8FlpF+JvD3wHUR8Vy2z/XAi1nb6gbgT7L15wIPA/uA1cDfRcQjVfsFrWn5y1lmVSCphbRVc01ErMq7HrPh+EjfbJwkLZU0K2vTfI20TbMu57LMRuTQNxu/hPRc+V2k3w34WNauMatbbu+YmRWIj/TNzAqk7i64dsopp0RXV1feZZiZNZQNGzbsiojRThuuv9Dv6uqip6cn7zLMzBqKpBcrGef2jplZgTj0zcwKxKFvZlYgDn0zswJx6JuZFYhD38ysQBz6ZmYF0jShv3s33HorPP746GPNzIqq7r6cNV6trfDNb8KRI7BgQd7VmJnVp6Y50p85Ey68EB7xbSTMzIbVNKEPkCSwZg0MDORdiZlZfWq60N+/H558Mu9KzMzqU1OF/uLF6U+3eMzMymuq0O/shK4uh76Z2XCaKvQhbfE88gj4hmBmZsdqytDfsQO2bMm7EjOz+tOUoQ9u8ZiZldN0oX/eeTB7tkPfzKycpgv9KVPSs3hWrcq7EjOz+tN0oQ9pi2fzZti5M+9KzMzqS0WhL2mppM2SeiXdXGb75ZIelzQg6ZqS9RdJWiNpk6SnJH2ymsUPZ7Cvv3p1Ld7NzKxxjBr6klqA24GrgPnAtZLmDxn2EvAZ4N4h698E/jQizgeWAn8r6cSJFj2a7m5ob3df38xsqEqusrkQ6I2ILQCSVgDLgKcHB0TEC9m2I6U7RsRzJY+3S3oV6ABen3DlI2hvh0suceibmQ1VSXtnDrC15Hlftm5MJC0EpgLPj3Xf8UgS2LAB3nyzFu9mZtYYKgl9lVk3pu+7Sjod+BHwZxFxpMz2GyX1SOrZWaVPX5MkvdrmunVVeTkzs6ZQSej3AXNLnncC2yt9A0kzgZ8D/yUiHis3JiLuiIjuiOju6Oio9KVHdNll6U+3eMzMjqok9NcD50g6U9JUYDmwspIXz8b/FLg7Iv5x/GWO3ezZcMEFDn0zs1Kjhn5EDAA3AQ8BzwD3R8QmSbdIuhpA0iWS+oBPAN+TtCnb/d8DlwOfkfTbbLloUn6TMpYsgUcfhcOHa/WOZmb1TVFnl6Ps7u6Onp6eqrzWvffCpz4FTzwBF9VsqjEzqz1JGyKie7RxTfmN3EG++JqZ2ds1dejPmwdz5zr0zcwGNXXoQ3q0v2qVb6piZgYFCf3t2+HFF/OuxMwsf4UIfXCLx8wMChD6558Ps2Y59M3MoACh39KSfjvXoW9mVoDQh7TFs2kT7N6ddyVmZvkqTOiDb6piZlaI0L/kEmhrc4vHzKwQoT9tWno3LYe+mRVdIUIf0hbP+vVw4EDelZiZ5adQod/fD1W6lpuZWUMqTOj7pipmZgUK/VNOgfPOc+ibWbEVJvQhbfGsXg1HjrlLr5lZMRQu9PfsSb+oZWZWRIUK/SVL0p9u8ZhZURUq9Lu64IwzHPpmVlyFCn0pbfE49M2sqAoV+pCG/ksvpYuZWdEUMvTBF18zs2IqXOi/5z0wY4ZbPGZWTIUL/dZWWLTIoW9mxVS40Ie0xbNxI7z+et6VmJnVVmFDPwLWrMm7EjOz2ipk6C9cmLZ5Vq3KuxIzs9qqKPQlLZW0WVKvpJvLbL9c0uOSBiRdM2Tb9ZL+X7ZcX63CJ2L6dFiwwH19MyueUUNfUgtwO3AVMB+4VtL8IcNeAj4D3Dtk35OAbwCXAguBb0iaPfGyJy5JYN06OHgw70rMzGqnkiP9hUBvRGyJiEPACmBZ6YCIeCEingKGXr/yw8AvI2J3RLwG/BJYWoW6JyxJ0sDfsCHvSszMaqeS0J8DbC153petq0RF+0q6UVKPpJ6dO3dW+NITs3hx+tMtHjMrkkpCX2XWRYWvX9G+EXFHRHRHRHdHR0eFLz0xp54K73qXQ9/MiqWS0O8D5pY87wS2V/j6E9l30vmmKmZWNJWE/nrgHElnSpoKLAdWVvj6DwFXSpqdfYB7ZbauLiQJ7N4Nzz6bdyVmZrUxauhHxABwE2lYPwPcHxGbJN0i6WoASZdI6gM+AXxP0qZs393AraQTx3rglmxdXRi8+JpbPGZWFIqotD1fG93d3dHT01OT94qAd7wDPvxhuPvumrylmdmkkLQhIrpHG1fIb+QOktJbKPpI38yKotChD2mL5/e/h23b8q7EzGzyOfR9UxUzK5DCh/5FF6XX4nGLx8yKoPCh39oK73ufQ9/MiqHwoQ9pi+fJJ+GNN/KuxMxscjn0SUP/yBF47LG8KzEzm1wOfeDSS6GlxS0eM2t+Dn1gxoz0A12Hvpk1O4d+JknS9s6hQ3lXYmY2eRz6mSSBAwfgiSfyrsTMbPI49DO+qYqZFYFDP3P66XDWWQ59M2tuDv0SSZKGfp1deNTMrGoc+iWSBHbtgueey7sSM7PJ4dAv4ZuqmFmzc+iXOPdcOPlkh76ZNS+HfgnpaF/fzKwZOfSHSBLo7YUdO/KuxMys+hz6QyxZkv70TVXMrBk59Ie4+GKYNs0tHjNrTg79IaZOTa+66dA3s2bk0C8jSdJr8Ozbl3clZmbV5dAvI0ng8GFYuzbvSszMqsuhX8aiRTBlils8ZtZ8HPplzJwJF17o0Dez5lNR6EtaKmmzpF5JN5fZ3i7pvmz7Wkld2fo2SXdJ2ijpGUlfrW75kydJYM0aGBjIuxIzs+oZNfQltQC3A1cB84FrJc0fMuwG4LWIOBu4DfhOtv4TQHtEvAd4L/C5wQmh3iUJ7N8PTz6ZdyVmZtVTyZH+QqA3IrZExCFgBbBsyJhlwF3Z4weAKyQJCGC6pFZgGnAIeKMqlU+ywZuqrFqVbx1mZtVUSejPAbaWPO/L1pUdExEDwB7gZNIJYD/wMvAS8N8iYvfQN5B0o6QeST07d+4c8y8xGTo7oavLfX0zay6VhL7KrBt6m5HhxiwEDgNnAGcCX5b0zmMGRtwREd0R0d3R0VFBSbXhm6qYWbOpJPT7gLklzzuB7cONyVo5s4DdwHXAP0dEf0S8CqwGuidadK0kCbzyCjz/fN6VmJlVRyWhvx44R9KZkqYCy4GVQ8asBK7PHl8DPBwRQdrS+YBS04H3Ac9Wp/TJ55uqmFmzGTX0sx79TcBDwDPA/RGxSdItkq7Oht0JnCypF/gSMHha5+3ACcDvSCePH0bEU1X+HSbNeefB7NkOfTNrHq2VDIqIB4EHh6z7esnjt0hPzxy6375y6xvFlCnpWTwOfTNrFv5G7iiSBDZvhjo5qcjMbEIc+qMY7Ov7pipm1gwc+qPo7ob2drd4zKw5OPRH0d4OCxc69M2sOTj0K5AksGEDvPlm3pWYmU2MQ78CSZJebXPdurwrMTObGId+BRYtAsktHjNrfA79CsyeDRdc4NA3s8bn0K9QksCjj6b3zjUza1QO/QolCezdCxs35l2Jmdn4OfQr5IuvmVkzcOhXaN48mDvXd9Iys8bm0B8D31TFzBqdQ38MkgS2b4cXXsi7EjOz8XHoj4H7+mbW6Bz6Y3D++TBrlkPfzBqXQ38MWlrgsssc+mbWuBz6Y5Qk8PTT8Ic/5F2JmdnYOfTHaLCv/+ij+dZhZjYeDv0xuuQSaGtzi8fMGpNDf4ymTUvvpuXQN7NG5NAfhySB9evhwIG8KzEzGxuH/jgsWQL9/dDTk3clZmZj49Afh8suS3+6xWNmjcahPw4nnwzz5zv0zazxOPTHKUlg9Wo4ciTvSszMKldR6EtaKmmzpF5JN5fZ3i7pvmz7WkldJdsulLRG0iZJGyUdV73y85MksGcPbNqUdyVmZpUbNfQltQC3A1cB84FrJc0fMuwG4LWIOBu4DfhOtm8r8GPgP0TE+cD7gf6qVZ8jX3zNzBpRJUf6C4HeiNgSEYeAFcCyIWOWAXdljx8ArpAk4ErgqYh4EiAi/hARTXGX2a4uOOMMh76ZNZZKQn8OsLXkeV+2ruyYiBgA9gAnA+8CQtJDkh6X9Ofl3kDSjZJ6JPXs3LlzrL9DLqSjN1UxM2sUlYS+yqwbeu+o4ca0AgnwqeznxyVdcczAiDsiojsiujs6OiooqT4kCbz0UrqYmTWCSkK/D5hb8rwT2D7cmKyPPwvYna3/TUTsiog3gQeBBRMtul64r29mjaaS0F8PnCPpTElTgeXAyiFjVgLXZ4+vAR6OiAAeAi6UdHw2GfwR8HR1Ss/fe94DM2Y49M2scbSONiAiBiTdRBrgLcAPImKTpFuAnohYCdwJ/EhSL+kR/vJs39ckfZd04gjgwYj4+ST9LjXX2gqLFjn0zaxxKD0grx/d3d3R00AXtbn1VvjGN9KbqsyenXc1ZlZUkjZERPdo4/yN3AlKEoiANWvyrsTMbHQO/QlauDBt87jFY2aNwKE/QdOnw4IFDn0zawwO/SpIEli3Dg4ezLsSM7OROfSrIEnSwN+wIe9KzMxG5tCvgsWL059u8ZhZvXPoV8Gpp8K55zr0zaz+OfSrxDdVMbNG4NCvkiSB3bvh2WfzrsTMbHgO/SrxxdfMrBE49KvkrLPgtNMc+mZW3xz6VeKbqphZI3DoV1GSwO9/D9u25V2JmVl5Dv0qcl/fzOqdQ7+KLroovRaPQ9/M6pVDv4paW+F973Pom1n9cuhXWZLAU0/Bnj15V2JmdiyHfpUlSfqt3Mcey7sSM7NjOfSr7NJLoaXFLR4zq08O/SqbMSP9QNehb2b1yKE/CZIE1q6FQ4fyrsTM7O0c+pMgSeDAAXjiibwrMTN7O4f+JPBNVcysXjn0J8Hpp6cXYHPom1m9cehPksGLr0XkXYmZ2VEO/UmyZAns2gXPPZd3JWZmR1UU+pKWStosqVfSzWW2t0u6L9u+VlLXkO3zJO2T9J+rU3b988XXzKwejRr6klqA24GrgPnAtZLmDxl2A/BaRJwN3AZ8Z8j224BfTLzcxvGud8Eppzj0zay+VHKkvxDojYgtEXEIWAEsGzJmGXBX9vgB4ApJApD0MWALsKk6JTcG31TFzOpRJaE/B9ha8rwvW1d2TEQMAHuAkyVNB/4C+NZIbyDpRkk9knp27txZae11L0mgtxd27Mi7EjOzVCWhrzLrhp6TMtyYbwG3RcS+kd4gIu6IiO6I6O7o6KigpMYw2NdfvTrfOszMBlUS+n3A3JLnncD24cZIagVmAbuBS4H/KukF4AvAX0q6aYI1N4yLL4Zp09ziMbP60VrBmPXAOZLOBLYBy4HrhoxZCVwPrAGuAR6OiACWDA6Q9E1gX0T8QxXqbghTp6ZX3Vy1Ku9KzMxSox7pZz36m4CHgGeA+yNik6RbJF2dDbuTtIffC3wJOOa0zqJKkvQaPHv35l2JmVllR/pExIPAg0PWfb3k8VvAJ0Z5jW+Oo76GN3hTlbVr4YMfzLsaMys6fyN3ki1aBFOmuK9vZvXBoT/JZs6ECy906JtZfXDo10CSpPfM7e/PuxIzKzqHfg0kCezfD08+mXclZlZ0Dv0a8E1VzKxeOPRroLMTuroc+maWP4d+jfimKmZWDxz6NZIk8Mor8PzzeVdiZkXm0K8R31TFzOqBQ79GzjsPTjrJoW9m+XLo18iUKelZPA59M8uTQ7+GkgQ2b4Ymuk+MmTUYh34N+aYqZpY3h34Nvfe90N7uFo+Z5cehX0Pt7bBwoUPfzPLj0K+xJIENG9Jr8ZiZ1ZpDv8aSBAYGYN26vCsxsyJy6NfYokUgucVjZvlw6NfY7NlwwQUOfTPLh0M/B0kCjz6atnnMzGrJoZ+DJIF9+2DjxrwrMbOicejnwBdfM7O8OPRzMG8ezJ0Lv/6175trZrXVmncBRXX55XDPPekXtk47LZ0EOjvLL3PmpOPMzCbKoZ+Tv/kbeP/7oa/v6PLcc/Dww7Bnz7HjOzqOTgLlJog5c+D442v+a5hZg3Ho5+S00+Czny2/be9e2Lbt7RNCXx9s3QovvphesG337mP3O+mkYyeDoRPECSdM7u9lZvXNoV+HZsyAd787XYbz5pvlJ4bBpacHXn312P1mzRq+jTQ4QcycmX6BzMyaT0WhL2kp8HdAC/D9iPj2kO3twN3Ae4E/AJ+MiBckfQj4NjAVOAR8JSIermL9hXX88XDOOekynIMHR54YnnoKduw49mbtJ5xQvn00e3Y6IcycmU5MpY/b2ib39zWz6hg19CW1ALcDHwL6gPWSVkbE0yXDbgBei4izJS0HvgN8EtgFfDQitku6AHgImFPtX8LKa2+Hd74zXYbT3w8vv1x+Uti6FX71K9i+HY4cGfm9jjuu/GQw3LrhHp9wArS0VPffwcyOquRIfyHQGxFbACStAJYBpaG/DPhm9vgB4B8kKSKeKBmzCThOUntEHJxw5VYVbW3pKaTz5g0/ZmAgbRXt2QNvvJEue/eWf1z6fNs2ePbZo+vfequymqZPn9jEMTh5HH+8JxCzoSoJ/TnA1pLnfcClw42JiAFJe4CTSY/0B/074IlygS/pRuBGgHkjpY/lorUVzjgjXSaiv//ohDDcpDHc41273j6xVHoJi+OOSyeR6dPTiWDw8XBLpWM8oVijqiT0y32kF2MZI+l80pbPleXeICLuAO4A6O7uHvra1iTa2tIzjE46aWKvE5F+XjHahLF///DL9u3HrhvrF+VKJ5TxTh6Dj9vb09drb3/70upTLazKKvkv1QfMLXneCWwfZkyfpFZgFrAbQFIn8FPgTyPi+QlXbIUnpQF53HFw6qnVe93+/mMngn37Rp48yo3ZsePYMYcOja+mKVOGnxBGWj9Z+0zxd/gbXiWhvx44R9KZwDZgOXDdkDErgeuBNcA1wMMREZJOBH4OfDUifDtwq2ttbXDiielSbeUmlNLlrbfSv17KLSNtO3gQDhyA118fef+hZ2iNV2vrsZPDWB5Xa+zUqZ6AxmvU0M969DeRnnnTAvwgIjZJugXoiYiVwJ3AjyT1kh7hL892vwk4G/iapK9l666MiDJnkJs1r8mcUEYTkX4GMp5Jpdy2oetKn7/11tEP7YfbXq1Lire1VTZBtLUdu0ydWp31Y92nHj4HUlTrEKBKuru7o6enJ+8yzGySHD5cftIYaaIYy7qh2/v7374cOnTsuv7+0U9LrgZp5IliwQJYsWK8r60NEdE92jh/TGRmNdXSkp79VG/Xijp8uPxkMNJEMdK28ewz0ndqqsWhb2ZGOhm1tKQtoWbmj0LMzArEoW9mViAOfTOzAnHom5kViEPfzKxAHPpmZgXi0DczKxCHvplZgdTdZRgk7QRenMBLnMLbr+NfzxqpVmisehupVmisehupVmiseidS67+JiI7RBtVd6E+UpJ5Krj9RDxqpVmisehupVmisehupVmisemtRq9s7ZmYF4tA3MyuQZgz9O/IuYAwaqVZorHobqVZorHobqVZorHonvdam6+mbmdnwmvFI38zMhuHQNzMrkKYJfUlLJW2W1Cvp5rzrGYmkH0h6VdLv8q5lNJLmSvq1pGckbZL0+bxrGomk4yStk/RkVu+38q5pNJJaJD0h6Wd51zIaSS9I2ijpt5Lq+r6mkk6U9ICkZ7P/v4vyrmk4ks7N/k0HlzckfWFS3qsZevqSWoDngA8BfcB64NqIeDrXwoYh6XJgH3B3RFyQdz0jkXQ6cHpEPC5pBrAB+Fgd/9sKmB4R+yS1AY8An4+Ix3IubViSvgR0AzMj4iN51zMSSS8A3RFR9192knQXsCoivi9pKnB8RLyed12jyfJsG3BpREzki6plNcuR/kKgNyK2RMQhYAWwLOeahhUR/wrszruOSkTEyxHxePZ4L/AMMCffqoYXqX3Z07ZsqdsjG0mdwB8D38+7lmYiaSZwOXAnQEQcaoTAz1wBPD8ZgQ/NE/pzgK0lz/uo42BqVJK6gIuBtflWMrKsXfJb4FXglxFRz/X+LfDnwJG8C6lQAP9H0gZJN+ZdzAjeCewEfpi1zr4vaXreRVVoOfCTyXrxZgl9lVlXt0d3jUjSCcA/AV+IiDfyrmckEXE4Ii4COoGFkuqyhSbpI8CrEbEh71rGYHFELACuAv5j1qqsR63AAuB/RMTFwH6grj/rA8jaUFcD/zhZ79Esod8HzC153glsz6mWppP1xv8JuCci/mfe9VQq+3P+X4ClOZcynMXA1VmffAXwAUk/zrekkUXE9uznq8BPSVur9agP6Cv5K+8B0kmg3l0FPB4Rr0zWGzRL6K8HzpF0ZjZTLgdW5lxTU8g+GL0TeCYivpt3PaOR1CHpxOzxNOCDwLP5VlVeRHw1Ijojoov0/+zDEfHpnMsalqTp2Yf5ZK2SK4G6PAMtInYAWyWdm626AqjLkw+GuJZJbO1A+idQw4uIAUk3AQ8BLcAPImJTzmUNS9JPgPcDp0jqA74REXfmW9WwFgN/AmzM+uQAfxkRD+ZY00hOB+7KzoCYAtwfEXV/KmSDOA34aXocQCtwb0T8c74ljeg/AfdkB4JbgD/LuZ4RSTqe9AzEz03q+zTDKZtmZlaZZmnvmJlZBRz6ZmYF4tA3MysQh76ZWYE49M3MCsShb2ZWIA59M7MC+f+/HqQ1vnNulQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGTtJREFUeJzt3X2QHPV95/H3h9UT4kFAtLJBTysezoeMZUm1FiGj465iB4vEkUgVrhMUKbjiglMVrhw75ztyqZicyB9OXBWcXHF3ULZTXGIQQknqdDY+jvNTLIsHrR5skBQcIethEZi1ZQsJCELie39072m0mt3p2Z3Z7un+vKq6dqb71zPfFdSne3/T/R1FBGZmVg3n5F2AmZlNHoe+mVmFOPTNzCrEoW9mViEOfTOzCnHom5lViEPfSk1Sj6Tjkha0c6xZt5Kv07cikXS87ulM4G3gVPr8ExHxlcmvauIk/TEwLyLuyLsWq7YpeRdgVi8izh9+LGk/8G8j4v+ONl7SlIg4ORm1mZWBp3esq0j6Y0mPSXpU0jHgNknXSXpG0s8lvSLpLyRNTcdPkRSS+tLnf51u/7qkY5KelrSo1bHp9hsl/VDSUUn/RdL3JN0xjt/p/ZK+k9b/vKRfq9v2MUl70vcflPSpdP0cSU+k+xyR9Pfj/Te1anHoWzf6DeARYBbwGHAS+CQwG6gBq4BPjLH/rcAfApcAB4H7Wh0raQ6wAfhM+r4/Ala0+otImgZ8Ffga0At8CnhM0pXpkL8E7oyIC4AlwHfS9Z8B9qX7vDet0awph751o80R8b8i4t2IeCsitkbEsxFxMiL2AQ8B/3KM/TdGxEBEvAN8BVg6jrEfA3ZGxP9Mt90P/GQcv0sNmAZ8PiLeSaeyvg6sTbe/AyyWdEFEHImI7XXrLwMWRMSJiPjOWa9s1oBD37rRofonkv65pK9JelXS68A6krPv0bxa9/hN4PzRBo4x9rL6OiK5ImIwQ+0jXQYcjDOvqDgAzE0f/wawGjgo6duSrk3Xfy4d9w1JL0n6zDje2yrIoW/daOQlZw8CLwBXRsSFwGcBdbiGV4B5w08kidNB3YrDwPx0/2ELgJcB0r9gVgNzSKaB1qfrX4+IT0VEH3AT8B8ljfXXjRng0LdyuAA4Crwh6WrGns9vl68CyyX9uqQpJJ8p9DbZp0fSjLplOrCF5DOJ35M0VdIvA78KbJB0rqRbJV2YTiEdI718NX3fK9KDxdF0/anGb2t2mkPfyuD3gNtJQvFBkg93Oyoifgz8a+DPgJ8CVwA7SO4rGM1twFt1y4sR8Tbw68Aaks8E/gK4NSJ+mO5zO3Agnba6E/jNdP37gG8Cx4HvAX8eEZvb9gtaafnmLLM2kNRDMlVzc0R8N+96zEbjM32zcZK0StKsdJrmD0mmaZ7LuSyzMTn0zcZvJcm18j8huTfgpnS6xqywPL1jZlYhPtM3M6uQwjVcmz17dvT19eVdhplZV9m2bdtPIqLZZcPFC/2+vj4GBgbyLsPMrKtIOpBlnKd3zMwqxKFvZlYhDn0zswpx6JuZVYhD38ysQhz6ZmYV4tA3M6uQ0oT+kSOwbh1s3958rJlZVRXu5qzx6umBP/ojiIDly/OuxsysmEpzpj9rFixZAt/7Xt6VmJkVV2lCH6BWg2eegVP+0jgzs4ZKF/rHjsHzz+ddiZlZMZUu9MFTPGZmoylV6C9YAHPnOvTNzEZTqtCXkrP9zZvzrsTMrJhKFfqQhP6hQ8liZmZnKmXog6d4zMwaKV3of/CDcN55Dn0zs0ZKF/pTpsC11zr0zcwaKV3oQzLF8/3vJ9fsm5nZaZlCX9IqSS9K2ivpngbbPy1pt6QfSPqGpIV1205J2pkum9pZ/GhqNXj3XXj22cl4NzOz7tE09CX1AA8ANwKLgVskLR4xbAfQHxFLgI3An9ZteysilqbL6jbVPaZf/MXk8k1P8ZiZnSnLmf4KYG9E7IuIE8B6YE39gIj4VkS8mT59BpjX3jJbM2sWfOADDn0zs5GyhP5coP6q98F03WjuBL5e93yGpAFJz0i6qdEOku5KxwwMDQ1lKKk5N18zMztbltBXg3XRcKB0G9APfL5u9YKI6AduBb4g6YqzXizioYjoj4j+3t7eDCU15+ZrZmZnyxL6g8D8uufzgMMjB0n6CPAHwOqIeHt4fUQcTn/uA74NLJtAvZn5Ji0zs7NlCf2twFWSFkmaBqwFzrgKR9Iy4EGSwH+tbv3Fkqanj2cDNWB3u4ofy8KFcNllDn0zs3pNvy4xIk5Kuht4EugBvhwRuyStAwYiYhPJdM75wOOSAA6mV+pcDTwo6V2SA8znImJSQn+4+ZpD38zstEzfkRsRTwBPjFj32brHHxllvy3AByZS4ETUavD44zA4CPNyvZ7IzKwYSnlH7jDP65uZnanUof/BD8LMmQ59M7NhpQ79qVPdfM3MrF6pQx9g5cqk+drx43lXYmaWv9KHfq2W3JXr5mtmZhUIfTdfMzM7rfSh7+ZrZmanlT70IZniefppN18zM6tM6Lv5mplZhUIfPMVjZlaJ0HfzNTOzRCVC383XzMwSlQh9SEL/4MGk+ZqZWVVVKvTBZ/tmVm2VCX03XzMzq1Dou/mamVmFQh+SKR43XzOzKqtc6Lv5mplVWaVC/7rr3HzNzKqtUqE/axZcc41D38yqq1KhD26+ZmbVVsnQP3YMXngh70rMzCZfJUMfPMVjZtVUudDv64NLL3Xom1k1VS703XzNzKqscqEPsHIlHDgAL7+cdyVmZpOrkqHveX0zq6pKhr6br5lZVVUy9Iebr23enHclZmaTK1PoS1ol6UVJeyXd02D7pyXtlvQDSd+QtLBu2+2S/jFdbm9n8RPh5mtmVkVNQ19SD/AAcCOwGLhF0uIRw3YA/RGxBNgI/Gm67yXAvcC1wArgXkkXt6/88XPzNTOroixn+iuAvRGxLyJOAOuBNfUDIuJbEfFm+vQZYF76+KPAUxFxJCJ+BjwFrGpP6RPj5mtmVkVZQn8ucKju+WC6bjR3Al9vZV9Jd0kakDQwNDSUoaSJc/M1M6uiLKGvBuui4UDpNqAf+Hwr+0bEQxHRHxH9vb29GUpqDzdfM7OqyRL6g8D8uufzgMMjB0n6CPAHwOqIeLuVffPi5mtmVjVZQn8rcJWkRZKmAWuBTfUDJC0DHiQJ/NfqNj0J3CDp4vQD3BvSdYXgm7TMrGqahn5EnATuJgnrPcCGiNglaZ2k1emwzwPnA49L2ilpU7rvEeA+kgPHVmBduq4Q3HzNzKpGEQ2n53PT398fAwMDk/Z+H/84bN0K+/dP2luambWdpG0R0d9sXCXvyK1Xq7n5mplVh0Pf8/pmViGVD/2lS918zcyqo/KhP3UqrFjh0Dezaqh86EMyxbNzp5uvmVn5OfQ53XztuefyrsTMrLMc+rj5mplVh0MfuOgieP/7HfpmVn4O/ZSbr5lZFTj0UytXwuuvw65deVdiZtY5Dv2Ub9Iysypw6KfcfM3MqsChn5KSs/3Nm/OuxMyscxz6ddx8zczKzqFfx/P6ZlZ2Dv06br5mZmXn0K/j5mtmVnYO/RHcfM3MysyhP4Kbr5lZmTn0R3DzNTMrM4f+CG6+ZmZl5tBvwM3XzKysHPoN1GpuvmZm5eTQb8A3aZlZWTn0G1i0CN77Xoe+mZWPQ7+B4eZrDn0zKxuH/ihqNdi/Hw4fzrsSM7P2ceiPwvP6ZlZGmUJf0ipJL0raK+meBtuvl7Rd0klJN4/YdkrSznTZ1K7CO23ZMjj3XIe+mZXLlGYDJPUADwC/AgwCWyVtiojddcMOAncA/77BS7wVEUvbUOukcvM1MyujLGf6K4C9EbEvIk4A64E19QMiYn9E/AB4twM15qZWgx074I038q7EzKw9soT+XOBQ3fPBdF1WMyQNSHpG0k0tVZczN18zs7LJEvpqsC5aeI8FEdEP3Ap8QdIVZ72BdFd6YBgYGhpq4aU7y83XzKxssoT+IDC/7vk8IPOFjBFxOP25D/g2sKzBmIcioj8i+nt7e7O+dMddfLGbr5lZuWQJ/a3AVZIWSZoGrAUyXYUj6WJJ09PHs4EasHvsvYqlVoMtW9x8zczKoWnoR8RJ4G7gSWAPsCEidklaJ2k1gKQPSRoEPg48KGm4VdnVwICk7wPfAj434qqfwnPzNTMrk6aXbAJExBPAEyPWfbbu8VaSaZ+R+20BPjDBGnNVf5PWkiX51mJmNlG+I7cJN18zszJx6Dfh5mtmViYO/QzcfM3MysKhn4Gbr5lZWTj0M3DzNTMrC4d+Bm6+ZmZl4dDPyM3XzKwMHPoZufmamZWBQz+j665LfnqKx8y6mUM/IzdfM7MycOi3oFaDp5+Gd0v1VTFmViUO/RbUanD0qJuvmVn3cui3wDdpmVm3c+i34PLL4T3vceibWfdy6LfAzdfMrNs59FtUq8GPfgSvvJJ3JWZmrXPot8jz+mbWzRz6LXLzNTPrZg79Fk2b5uZrZta9HPrjUKvB9u1uvmZm3cehPw5uvmZm3cqhPw5uvmZm3cqhPw5uvmZm3cqhP05uvmZm3cihP05uvmZm3cihP06+ScvMupFDf5zcfM3MupFDf5zcfM3MupFDfwLcfM3Muk2m0Je0StKLkvZKuqfB9uslbZd0UtLNI7bdLukf0+X2dhVeBJ7XN7Nu0zT0JfUADwA3AouBWyQtHjHsIHAH8MiIfS8B7gWuBVYA90q6eOJlF8OyZTBjhkPfzLpHljP9FcDeiNgXESeA9cCa+gERsT8ifgCMvGr9o8BTEXEkIn4GPAWsakPdheDma2bWbbKE/lzgUN3zwXRdFpn2lXSXpAFJA0NDQxlfuhhqNdixA958M+9KzMyayxL6arAuMr5+pn0j4qGI6I+I/t7e3owvXQy1Gpw86eZrZtYdsoT+IDC/7vk84HDG15/Ivl3BzdfMrJtkCf2twFWSFkmaBqwFNmV8/SeBGyRdnH6Ae0O6rjQuuQQWL3bom1l3aBr6EXESuJskrPcAGyJil6R1klYDSPqQpEHg48CDknal+x4B7iM5cGwF1qXrSsXN18ysWygi6/T85Ojv74+BgYG8y2jJww/DHXfA88/DNdfkXY2ZVZGkbRHR32yc78htA9+kZWbdwqHfBldc4eZrZtYdHPpt4OZrZtYtHPptUqvBvn1uvmZmxebQbxPP65tZN3Dot4mbr5lZN3Dot4mbr5lZN3Dot5Gbr5lZ0Tn028jN18ys6Bz6beTma2ZWdA79NnLzNTMrOod+m7n5mpkVmUO/zWo1+PnPYffuvCsxMzubQ7/NfJOWmRWZQ7/NrrgC5sxx6JtZMTn028zN18ysyBz6HTDcfO3VV/OuxMzsTA79DvC8vpkVlUO/A5Yvd/M1Mysmh34HTJsGH/qQQ9/Miseh3yG1Gmzf7uZrZlYsDv0OGW6+tnVr3pWYmZ3m0O+QX/ql5KeneMysSBz6HXLJJXD11Q59MysWh34HrVwJW7a4+ZqZFYdDv4PcfM3Misah30G+ScvMisah30FuvmZmRePQ7yA3XzOzoskU+pJWSXpR0l5J9zTYPl3SY+n2ZyX1pev7JL0laWe6/Pf2ll98br5mZkXSNPQl9QAPADcCi4FbJC0eMexO4GcRcSVwP/Anddteioil6fLbbaq7a3he38yKJMuZ/gpgb0Tsi4gTwHpgzYgxa4CH08cbgQ9LUvvK7F5uvmZmRZIl9OcCh+qeD6brGo6JiJPAUeAX0m2LJO2Q9B1J/6LRG0i6S9KApIGhoaGWfoGic/M1MyuSLKHf6Iw9Mo55BVgQEcuATwOPSLrwrIERD0VEf0T09/b2Ziipu7j5mpkVRZbQHwTm1z2fBxwebYykKcAs4EhEvB0RPwWIiG3AS8A/m2jR3cbN18ysKLKE/lbgKkmLJE0D1gKbRozZBNyePr4Z+GZEhKTe9INgJF0OXAXsa0/p3cPN18ysKKY0GxARJyXdDTwJ9ABfjohdktYBAxGxCfgS8FeS9gJHSA4MANcD6ySdBE4Bvx0RRzrxixSZm6+ZWVEoYuT0fL76+/tjYGAg7zLa7rd+CzZuhJ/+FM7xLXFm1maStkVEf7Nxjp9JMtx8bc+evCsxsypz6E8S36RlZkXg0J8kV14Jvb0OfTPLl0N/krj5mpkVgUN/EtVq8NJL8OMf512JmVWVQ38SeV7fzPLm0J9Ey5fD9OkOfTPLj0N/Ek2f7uZrZpYvh/4kW7kStm1z8zUzy4dDf5K5+ZqZ5cmhP8ncfM3M8uTQn2RuvmZmeXLo56BWgy1b4N13867EzKrGoZ8DN18zs7w49HPgm7TMLC8O/Ry4+ZqZ5aXpN2dZ+w03X3vkkeTSzYULk6Wv7/TjhQvh0kv9hStm1l4O/Zzcd18S8gcOJMvWrcm3atWbOhXmz298QFi4EObNg2nT8qjezLqVQz8n11wD999/5rrjx+HgQdi///TBYHh58kl45RWo/3ZLCS67rPEBYXiZOXMyfyszKzqHfoGcfz4sXpwsjbz9Nhw6dPYB4cABePpp2LAhudu3Xm/v6AeEvj646KKO/1pmViAO/S4yfXryIfCVVzbefuoUHD589gFh/3544QX42tfgn/7pzH0uvHD0A8LChTBnTvIXhZmVg0O/RHp6ks8A5s9PGruNFAFDQ40PCgcOwHe/C0ePnrnPjBmwYEGyXHQRnHfe2MvMmaNvmzbNBxCzvDn0K0RKztznzElaPDdy9GjjA8KhQ/Dyy/DGG6eXkX81NNPT0/ygMd6DysyZPqCYZeHQtzPMmgVLliRLM6dOJS2i6w8EWZf6/Y4dg1dfPXtM/YfWWdQfEM49N7n6aerU5C+M4cf1SyvrO/UaviTXJptD38atpwcuuCBZ2i0i+UtiPAeU4b9C3nknWU6cSH6+9Ra8/vrZ60cuw+tbPeiMxznnJOE/ZUqy9PQky/Djsda1Or5d63p6krqzLlJr49uxb/1+w4+lsR8PL2Xn0LdCkpKz9XPPhdmz86nh1KnsB4iJrD9xInmvU6eSq6/qf2Zdd+LE+Petf+wmgM0PDmMdNFrdZ+T+S5fCo4929vdz6JuNYvisdsaMvCuZPBHZDg4RyQEi69Lq+HbtO7z/8Gu08niy9ql/fPnlnf9v7NA3s/9POj3VNH163tVYJ2T6GEnSKkkvStor6Z4G26dLeizd/qykvrptv5+uf1HSR9tXupmZtapp6EvqAR4AbgQWA7dIGnnP6J3AzyLiSuB+4E/SfRcDa4H3A6uA/5q+npmZ5SDLmf4KYG9E7IuIE8B6YM2IMWuAh9PHG4EPS1K6fn1EvB0RPwL2pq9nZmY5yBL6c4FDdc8H03UNx0TESeAo8AsZ90XSXZIGJA0MDQ1lr97MzFqSJfQbXbk68grm0cZk2ZeIeCgi+iOiv7e3N0NJZmY2HllCfxCYX/d8HnB4tDGSpgCzgCMZ9zUzs0mSJfS3AldJWiRpGskHs5tGjNkE3J4+vhn4ZkREun5tenXPIuAq4Ln2lG5mZq1qep1+RJyUdDfwJNADfDkidklaBwxExCbgS8BfSdpLcoa/Nt13l6QNwG7gJPA7EXGqQ7+LmZk1oZiMBiMtkDQEHJjAS8wGftKmcjqtm2qF7qq3m2qF7qq3m2qF7qp3IrUujIimH4oWLvQnStJARPTnXUcW3VQrdFe93VQrdFe93VQrdFe9k1GrG7uamVWIQ9/MrELKGPoP5V1AC7qpVuiuerupVuiuerupVuiuejtea+nm9M3MbHRlPNM3M7NROPTNzCqkNKHfrOd/kUj6sqTXJL2Qdy3NSJov6VuS9kjaJemTedc0FkkzJD0n6ftpvf8575qakdQjaYekr+ZdSzOS9kt6XtJOSQN51zMWSRdJ2ijpH9L/f6/Lu6bRSHpf+m86vLwu6Xc78l5lmNNPe/T/EPgVkn4/W4FbImJ3roWNQtL1wHHgf0TENXnXMxZJlwKXRsR2SRcA24CbCvxvK+C8iDguaSqwGfhkRDyTc2mjkvRpoB+4MCI+lnc9Y5G0H+iPiMLf7CTpYeC7EfHFtIXMzIj4ed51NZPm2cvAtRExkRtVGyrLmX6Wnv+FERF/T9KuovAi4pWI2J4+PgbsoUF77KKIxPH06dR0KeyZjaR5wK8BX8y7ljKRdCFwPUmLGCLiRDcEfurDwEudCHwoT+hn6ttvE5N+DeYy4Nl8KxlbOl2yE3gNeCoiilzvF4D/ALybdyEZBfB/JG2TdFfexYzhcmAI+Mt06uyLks7Lu6iM1gKPdurFyxL6mfr22/hJOh/4G+B3I+L1vOsZS0ScioilJK28V0gq5BSapI8Br0XEtrxraUEtIpaTfH3q76RTlUU0BVgO/LeIWAa8ART6sz6AdBpqNfB4p96jLKHvvv0dlM6N/w3wlYj427zrySr9c/7bJN/PXEQ1YHU6T74e+GVJf51vSWOLiMPpz9eAv6O4X386CAzW/ZW3keQgUHQ3Atsj4sedeoOyhH6Wnv82DukHo18C9kTEn+VdTzOSeiVdlD4+F/gI8A/5VtVYRPx+RMyLiD6S/2e/GRG35VzWqCSdl36YTzpVcgNQyCvQIuJV4JCk96WrPkzS4r3obqGDUzuQoZ9+Nxit53/OZY1K0qPAvwJmSxoE7o2IL+Vb1ahqwG8Cz6fz5AD/KSKeyLGmsVwKPJxeAXEOsCEiCn8pZJd4D/B3yXkAU4BHIuJ/51vSmP4d8JX0RHAf8G9yrmdMkmaSXIH4iY6+Txku2TQzs2zKMr1jZmYZOPTNzCrEoW9mViEOfTOzCnHom5lViEPfzKxCHPpmZhXy/wBBIpmHALpx6wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGd1JREFUeJzt3X2QHPWd3/H3R7urJwQIrCUGPSAwAiMBRmJZUG/iSx0XLHI+waXsRFC+wilS2FUm5bOvnHBJnakIX9VdXHW+XIokULavHJ95Oi4XEw6HED/dWehpJfFgSUYWMqBFcEgIkIQEevrmj+6NhmV3p3d3drpn+vOq6prp7l/PfHdL+nTvd7p7FBGYmVk1TCm6ADMzax6HvplZhTj0zcwqxKFvZlYhDn0zswpx6JuZVYhD39qapA5JhyQtaORYs1Yln6dvZSLpUM3sTOA94EQ2/7mI+F7zq5o4SV8D5kXEZ4uuxaqts+gCzGpFxKzB55JeBP5VRPzfkcZL6oyI482ozawduL1jLUXS1yQ9JOkBSQeBz0haLmmdpLckvSrpzyR1ZeM7JYWkhdn8X2TrfyDpoKS1ki4Y69hs/Q2Sdkh6W9J/lrRG0mfH8TMtkfTTrP7nJP1mzbpPStqevf+ApC9ly8+R9Hi2zX5Jfzve36lVi0PfWtFvA/cDZwIPAceBLwJzgD5gBfC5Uba/BfgD4GzgZeDusY6VdA7wMPCV7H1/BfSO9QeRNBV4DPgboBv4EvCQpIuyIX8O3BYRpwNXAD/Nln8F2JVt8+GsRrO6HPrWin4WEf8rIk5GxJGI2BgR6yPieETsAu4Dfm2U7R+JiP6IOAZ8D7hyHGM/CTwdEd/P1n0D2DeOn6UPmAp8PSKOZa2sHwCrsvXHgMWSTo+I/RGxuWb5ecCCiDgaET/9wCubDcOhb61od+2MpI9K+htJr0k6AKwmPfoeyWs1zw8Ds0YaOMrY82rriPSMiIEctQ91HvByvP+MipeAudnz3wZWAi9L+omka7Llf5SN+6GkFyR9ZRzvbRXk0LdWNPSUs3uBnwMXRcQZwFcBTXINrwLzBmckiVNBPRZ7gPnZ9oMWAK8AZH/BrATOIW0DPZgtPxARX4qIhcBNwL+VNNpfN2aAQ9/aw+nA28A7ki5l9H5+ozwGLJP0W5I6ST9T6K6zTYek6TXTNOAp0s8kfk9Sl6RfB/4p8LCkGZJukXRG1kI6SHb6ava+H8l2Fm9ny08M/7Zmpzj0rR38HnAraSjeS/rh7qSKiL8H/gXwJ8AbwEeALaTXFYzkM8CRmun5iHgP+C3gRtLPBP4MuCUidmTb3Aq8lLWtbgN+J1t+CfAj4BCwBvhPEfGzhv2A1rZ8cZZZA0jqIG3VfCoi/q7oesxG4iN9s3GStELSmVmb5g9I2zQbCi7LbFQOfbPx+4ek58rvI7024KasXWNWWm7vmJlViI/0zcwqpHQ3XJszZ04sXLiw6DLMzFrKpk2b9kVEvdOGyxf6CxcupL+/v+gyzMxaiqSX8oxze8fMrEIc+mZmFeLQNzOrEIe+mVmFOPTNzCrEoW9mViEOfTOzCmmb0N+/H+6+GzZvrj/WzKyqSndx1nh1dMBdd8HJk7BsWdHVmJmVU9sc6Z95Jlx2GTz1VNGVmJmVV9uEPkCSwLp1cMJfGmdmNqy2Cv2+PjhwALZtK7oSM7NyaqvQT5L00S0eM7PhtVXoX3ghnHOOQ9/MbCRtFfpSerTv0DczG15bhT6kob9zJ7z+etGVmJmVT1uGPvho38xsOG0X+lddBV1dDn0zs+G0XehPn54Gv0PfzOyD2i70IT1fv78f3nuv6ErMzMqlLUM/SdLA37Kl6ErMzMqlLUN/+fL00S0eM7P3a8vQP/dcuOACh76Z2VBtGfqQtnjWrIGIoisxMyuPtg79116DF18suhIzs/Jo69AHt3jMzGq1behfdhnMmuXQNzOr1bah39kJ117r0Dczq5Ur9CWtkPS8pJ2S7hxm/ZclbZP0rKQfSjq/Zt2tkn6ZTbc2svh6kgSefRYOHmzmu5qZlVfd0JfUAdwD3AAsBm6WtHjIsC1AT0RcATwC/Mds27OBu4BrgF7gLklnNa780SVJ+kXpGzY06x3NzMotz5F+L7AzInZFxFHgQeDG2gER8eOIOJzNrgPmZc8/ATwZEfsj4k3gSWBFY0qv75pr0nvsu8VjZpbKE/pzgd018wPZspHcBvxgLNtKul1Sv6T+vXv35igpn9mzYckSh76Z2aA8oa9hlg17yZOkzwA9wNfHsm1E3BcRPRHR093dnaOk/JIE1q5N2zxmZlWXJ/QHgPk18/OAPUMHSfoN4N8DKyPivbFsO5mSBN5+G7Zta+a7mpmVU57Q3wgsknSBpKnAKuDR2gGSlgL3kgZ+7RcVPgFcL+ms7APc67NlTeOLtMzMTqkb+hFxHLiDNKy3Aw9HxFZJqyWtzIZ9HZgF/KWkpyU9mm27H7ibdMexEVidLWuaiy6C7m6HvpkZgKJkdyTr6emJ/v7+hr7mTTel7Z0dOxr6smZmpSFpU0T01BvXtlfk1koS+OUvoYEnBpmZtaTKhD6kZ/GYmVVZJUL/qqugq8t9fTOzSoT+jBmwbFn6pSpmZlVWidCHtMWzcSMcPVp0JWZmxalM6Pf1wXvvwZYtRVdiZlacyoT+8uXpo/v6ZlZllQn9886DhQsd+mZWbZUJfUj7+k89BSW7Hs3MrGkqF/p79sDLLxddiZlZMSoX+uAWj5lVV6VC//LL4bTTfL6+mVVXpUK/szP9CkUf6ZtZVVUq9CE9X/+ZZ+DQoaIrMTNrvsqFfpKkX524YUPRlZiZNV/lQv/aa9NHt3jMrIoqF/qzZ8OSJQ59M6umyoU+pC2etWvTNo+ZWZVUNvTfegt+8YuiKzEza67Khj74fH0zq55Khv6iRTBnjvv6ZlY9lQx96dTN18zMqqSSoQ9p6O/YAfv2FV2JmVnzVDr0IT2Lx8ysKiob+j096b143OIxsyqpbOjPmAHLljn0zaxaKhv6kLZ4NmyAY8eKrsTMrDkqH/rvvgtbthRdiZlZc1Q+9MEtHjOrjkqH/ty5cP75Dn0zq45Khz6kR/tr1kBE0ZWYmU0+h34Ce/bA7t1FV2JmNvkc+u7rm1mFVD70r7gCZs506JtZNVQ+9Ds74ZprHPpmVg2VD31IWzxPPw2HDhVdiZnZ5MoV+pJWSHpe0k5Jdw6z/uOSNks6LulTQ9adkPR0Nj3aqMIbqa8PTpyAjRuLrsTMbHLVDX1JHcA9wA3AYuBmSYuHDHsZ+Cxw/zAvcSQirsymlROsd1Jce2366BaPmbW7zhxjeoGdEbELQNKDwI3AtsEBEfFitq4lv2r8rLNg8WKHvpm1vzztnblA7VnsA9myvKZL6pe0TtJNY6quiZIkvbf+yZbcbZmZ5ZMn9DXMsrFcv7ogInqAW4A/lfSRD7yBdHu2Y+jfu3fvGF66cZIE3nwTnn++kLc3M2uKPKE/AMyvmZ8H7Mn7BhGxJ3vcBfwEWDrMmPsioicierq7u/O+dEP5Ii0zq4I8ob8RWCTpAklTgVVArrNwJJ0laVr2fA7QR81nAWVy8cVw9tkOfTNrb3VDPyKOA3cATwDbgYcjYquk1ZJWAki6WtIA8GngXklbs80vBfolPQP8GPijiChl6Eunbr5mZtau8py9Q0Q8Djw+ZNlXa55vJG37DN3uKeDyCdbYNH198NhjsG8fzJlTdDVmZo3nK3JrDPb1160rtg4zs8ni0K/R05Pei8d9fTNrVw79GjNnwtKlDn0za18O/SGSBDZsgGPHiq7EzKzxHPpDJAkcOQLPPFN0JWZmjefQH8IXaZlZO3PoDzFvHsyf7/P1zaw9OfSH0dfnI30za08O/WEkCQwMwO7d9ceambUSh/4w3Nc3s3bl0B/GFVek5+w79M2s3Tj0h9HVBb29Dn0zaz8O/REkCWzZAu+8U3QlZmaN49AfQZLAiRPQ3190JWZmjePQH8Hy5emjz9c3s3bi0B/B2WfDpZe6r29m7cWhP4okgbVr4eTJoisxM2sMh/4okgT274cdO4quxMysMRz6o/BFWmbWbhz6o7j44rS379A3s3bh0B/FlCnpWTwOfTNrFw79OpIEtm9Pe/tmZq3OoV9HX1/6uHZtsXWYmTWCQ7+Oq6+Gjg63eMysPTj065g5E5YudeibWXtw6OeQJLBhAxw7VnQlZmYT49DPIUng8GF49tmiKzEzmxiHfg6+SMvM2oVDP4f582HePIe+mbU+h35OSeLQN7PW59DPqa8PXn4ZBgaKrsTMbPwc+jm5r29m7cChn9PHPgYzZjj0zay1OfRz6uqC3l6Hvpm1Nof+GCQJbNmSnrNvZtaKHPpjkCRw/Dj09xddiZnZ+Dj0x+Daa9NHt3jMrFXlCn1JKyQ9L2mnpDuHWf9xSZslHZf0qSHrbpX0y2y6tVGFF2HOHLjkEoe+mbWuuqEvqQO4B7gBWAzcLGnxkGEvA58F7h+y7dnAXcA1QC9wl6SzJl52cfr60tCPKLoSM7Oxy3Ok3wvsjIhdEXEUeBC4sXZARLwYEc8CJ4ds+wngyYjYHxFvAk8CKxpQd2GSBN54A3bsKLoSM7OxyxP6c4HdNfMD2bI8cm0r6XZJ/ZL69+7dm/Oli+GLtMysleUJfQ2zLG9zI9e2EXFfRPRERE93d3fOly7GJZfAWWc59M2sNeUJ/QFgfs38PGBPztefyLalNGUKLF/u0Dez1pQn9DcCiyRdIGkqsAp4NOfrPwFcL+ms7APc67NlLS1JYNs2ePPNoisxMxubuqEfEceBO0jDejvwcERslbRa0koASVdLGgA+DdwraWu27X7gbtIdx0ZgdbaspQ329detK7YOM7Ox6swzKCIeBx4fsuyrNc83krZuhtv228C3J1Bj6fT2QkdH2uK54YaiqzEzy89X5I7DaafBlVfCmjVFV2JmNjYO/XFKEli/Pr0Xj5lZq3Doj1OSpHfbfPbZoisxM8vPoT9OvkjLzFqRQ3+c5s+HuXMd+mbWWhz64ySlR/sOfTNrJQ79CUgSeOkleOWVoisxM8vHoT8BfX3p49q1xdZhZpaXQ38CrrwSZszw+fpm1joc+hPQ1QVXX+2+vpm1Dof+BCUJbN4MR44UXYmZWX0O/QlKkvSq3P7+oisxM6vPoT9By5enj27xmFkrcOhP0Jw5cPHFDn0zaw0O/QYYvEgr8n6JpJlZQRz6DdDXB/v2wc6dRVdiZjY6h34DDN58zefrm1nZOfQb4KMfhdmz3dc3s/Jz6DfAlCnpWTwOfTMrO4d+gyQJbN0Kb71VdCVmZiNz6DfIYF9/3bpi6zAzG41Dv0F6e9M2j1s8ZlZmDv0GmTULPvYxh76ZlZtDv4H6+mD9+vRePGZmZeTQb6AkgUOH4Lnniq7EzGx4Dv0GGvww1y0eMysrh34DLVgA553n0Dez8nLoN5B06uZrZmZl5NBvsCSBF1+EPXuKrsTM7IMc+g022Ndfu7bYOszMhuPQb7ClS2H6dLd4zKycHPoNNnUqXH21Q9/MysmhPwmSBDZtgiNHiq7EzOz9HPqTIEng2LE0+M3MysShPwmWL08f3eIxs7Jx6E+C7m5YtMihb2blkyv0Ja2Q9LyknZLuHGb9NEkPZevXS1qYLV8o6Yikp7PpvzW2/PIavEgrouhKzMxOqRv6kjqAe4AbgMXAzZIWDxl2G/BmRFwEfAP445p1L0TEldn0+QbVXXpJAnv3wgsvFF2JmdkpeY70e4GdEbErIo4CDwI3DhlzI/Cd7PkjwHWS1LgyW09fX/roFo+ZlUme0J8L7K6ZH8iWDTsmIo4DbwMfytZdIGmLpJ9K+kcTrLdlXHopnHmmQ9/MyqUzx5jhjtiHdqpHGvMqsCAi3pB0FfA/JS2JiAPv21i6HbgdYMGCBTlKKr8pU9KzeNasKboSM7NT8hzpDwDza+bnAUNvJ/b/x0jqBM4E9kfEexHxBkBEbAJeAC4e+gYRcV9E9ERET3d399h/ipJKEti6Fd56q+hKzMxSeUJ/I7BI0gWSpgKrgEeHjHkUuDV7/ingRxERkrqzD4KRdCGwCNjVmNLLL0nSs3fWry+6EjOzVN3Qz3r0dwBPANuBhyNiq6TVklZmw74FfEjSTuDLwOBpnR8HnpX0DOkHvJ+PiP2N/iHKqrc3bfO4r29mZaEo2YnkPT090d/fX3QZDbN0KcyZA08+WXQlZtbOJG2KiJ5643xF7iRLEli3Dk6cKLoSMzOH/qTr64NDh+DnPy+6EjMzh/6kG/wmLff1zawMHPqT7Pzz4dxzfb6+mZWDQ3+SSaduvmZmVjSHfhMkCfzqV/Dqq0VXYmZV59BvgsG+/tq1xdZhZubQb4KlS2HaNLd4zKx4Dv0mmDYNenoc+mZWPId+k/T1pV+U/u67RVdiZlXm0G+SJIGjR2Hz5qIrMbMqc+g3yfLl6aPP1zezIjn0m+Scc+Cii9zXN7NiOfSbaPAirZLd2NTMKsSh30RJAq+/Drsq8zUyZlY2Dv0m8s3XzKxoDv0mWrIEzjjDoW9mxXHoN9GUKelZPA59MyuKQ7/JkgSeew4OHCi6EjOrIod+kyVJevbOunVFV2JmVeTQb7Le3rTN4xaPmRXBod9kZ5wBl1/u0DezYjj0C5AkaXvnxImiKzGzqnHoFyBJ4OBB2Lq16ErMrGoc+gXo60sf3eIxs2Zz6Bdg4UL48Icd+mbWfJ1FF1BFUtri+e534fvfh9NPTz/gHZxq50dbVzs/bVr6umZmo3HoF+RrX0tvy3DgQNrfP3Dg1LRnz/vn89yVs6tr/DuM2uennw6d/ldh1rb837sgl14Kq1fXHxcBhw+f2gEM3UGMNr93L7zwwqn5w4fz1TZz5vA7iM7O9K+JKVMm77FRr9XVBdOnw4wZ6c8zY8boU1eX/1KyanDol5wEp52WTueeO7HXOn4cDh0a+87j4EF46aX0FNOTJ9Md0WQ8jrSuGaZMqb9jGG3Ks2OpnaZP907GiuHQr5DOTpg9O51aScTYdyDHjsGRI42bDhxIHw8ffv/yY8fG/3MN/iUyY8apv1BaYershKlT07+Opk794DTc8kaN7epKf1c2fg59K73BsCnjf/YTJxqzUxncYZVlGq2e48fTnd3Ro+lU+7x22WQZ3Onk3ZkM7ixqn9ebH8vYsWzb0TF5v5fcv7+iCzBrZR0dMGtWOtkpgzuH4XYGQ5eNtuOY6Nh33knXDU6DY0ean2zS6DuIZcvggQcmtwaHvpk13OCH6V1d6edRraD2r5i8O4lGz1944eT/nA59MzPev6NqZyXskpqZ2WRx6JuZVUiu0Je0QtLzknZKunOY9dMkPZStXy9pYc2638+WPy/pE40r3czMxqpu6EvqAO4BbgAWAzdLWjxk2G3AmxFxEfAN4I+zbRcDq4AlwArgv2SvZ2ZmBchzpN8L7IyIXRFxFHgQuHHImBuB72TPHwGuk6Rs+YMR8V5E/ArYmb2emZkVIE/ozwV218wPZMuGHRMRx4G3gQ/l3BZJt0vql9S/d+/e/NWbmdmY5An94e4QMvSOKCONybMtEXFfRPRERE93d3eOkszMbDzyhP4AML9mfh6wZ6QxkjqBM4H9Obc1M7MmUdS5jWEW4juA64BXgI3ALRGxtWbMF4DLI+LzklYB/ywi/rmkJcD9pH3884AfAosiYsSvBJe0F3hpAj/THGDfBLZvplaqFVqr3laqFVqr3laqFVqr3onUen5E1G2V1L0iNyKOS7oDeALoAL4dEVslrQb6I+JR4FvAdyXtJD3CX5Vtu1XSw8A24DjwhdECP9tmQv0dSf0R0TOR12iWVqoVWqveVqoVWqveVqoVWqveZtSa6zYMEfE48PiQZV+tef4u8OkRtv1D4A8nUKOZmTWIr8g1M6uQdgz9+4ouYAxaqVZorXpbqVZorXpbqVZorXonvda6H+SamVn7aMcjfTMzG4FD38ysQtom9OvdCbRMJH1b0uuSfl50LfVImi/px5K2S9oq6YtF1zQaSdMlbZD0TFbvfyi6pnokdUjaIumxomupR9KLkp6T9LSk/qLrGY2k2ZIekfSL7N/v8qJrGomkS7Lf6eB0QNLvTsp7tUNPP7tz5w7gn5BeBbwRuDkithVa2AgkfRw4BPz3iLis6HpGI+lc4NyI2CzpdGATcFOJf7cCTouIQ5K6gJ8BX4yIdQWXNiJJXwZ6gDMi4pNF1zMaSS8CPRFR+oudJH0H+LuI+KakqcDMiHir6LrqyfLsFeCaiJjIharDapcj/Tx3Ai2NiPhb0ovYSi8iXo2Izdnzg8B2hrlpXllE6lA225VNpT2ykTQP+E3gm0XX0k4knQF8nPTCUSLiaCsEfuY64IXJCHxon9DPdTdPm5jsy3GWAuuLrWR0WbvkaeB14MmIKHO9fwr8G+Bk0YXkFMD/kbRJ0u1FFzOKC4G9wJ9nrbNvSmqRr2hnFfDAZL14u4R+rrt52vhJmgX8FfC7EXGg6HpGExEnIuJK0hv89UoqZQtN0ieB1yNiU9G1jEFfRCwj/VKlL2StyjLqBJYB/zUilgLvAKX+rA8ga0OtBP5yst6jXULfd/OcRFlv/K+A70XE/yi6nryyP+d/QvqtbWXUB6zM+uQPAr8u6S+KLWl0EbEne3wd+GvK+6VIA8BAzV95j5DuBMruBmBzRPz9ZL1Bu4T+RmCRpAuyPeUq4NGCa2oL2Qej3wK2R8SfFF1PPZK6Jc3Ons8AfgP4RbFVDS8ifj8i5kXEQtJ/sz+KiM8UXNaIJJ2WfZhP1iq5HijlGWgR8RqwW9Il2aLrSG/8WHY3M4mtHch5w7WyG+lOoAWXNSJJDwD/GJgjaQC4KyK+VWxVI+oDfgd4LuuTA/y77CZ8ZXQu8J3sDIgpwMMRUfpTIVvEPwD+Oj0OoBO4PyL+d7EljepfA9/LDgR3Af+y4HpGJWkm6RmIn5vU92mHUzbNzCyfdmnvmJlZDg59M7MKceibmVWIQ9/MrEIc+mZmFeLQNzOrEIe+mVmF/D8ALSGjU9b5TAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "### VISUALIZATION ###\n",
    "def training_plot(metrics):\n",
    "    plt.figure(1)\n",
    "    plt.plot([m.loss for m in metrics], 'b')\n",
    "    plt.title('Training Loss')\n",
    "    plt.show()\n",
    "\n",
    "training_plot(sgd_trainer.metrics)\n",
    "training_plot(adam_trainer.metrics)\n",
    "training_plot(rms_trainer.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter Initialization\n",
    "\n",
    "While training a network, the initial value of the weights plays a significant role. In the extreme case, an oracle could just set the weights directly to values that minimize the objective function, and in practical cases a good initialization can bring us to a more favorable starting position in the parameter space. \n",
    "\n",
    "This raises the question of how to choose these weights. \n",
    "\n",
    "- What happens if all the weights are set to zero? The gradients become zero, and the network finds itself without a direction. \n",
    "- What if all of them are set to the same non-zero value? Although the gradients are no longer zero, each neuron has the same weight and follows the same gradient. Such neurons will continue to have the same value, since they're identical. \n",
    "\n",
    "So any initialization scheme must break this symmetry somehow, and randomly initializing the weights is a first step in that direction.\n",
    "\n",
    "Let's begin with creating a weight initialization function that samples from **N(0,1)**. A clean way of initializing the weights is to access the network parameters by traversing all modules inside the network, and then applying the desired initialization. This method also allows us to encapsulate all the initializations into a single function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_randn(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        m.weight.data.normal_(0,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use this scheme to initialize the network.\n",
    "\n",
    "Note that *apply(fn)* applies the function *fn* recursively to every submodule (as returned by .children()) as well as self. Also, since it is applied to itself as well, you must take care to select the appropriate type of module *m* and apply the initialization to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FashionModel(\n",
       "  (fc1): Linear(in_features=784, out_features=64, bias=True)\n",
       "  (fc2): Linear(in_features=64, out_features=32, bias=True)\n",
       "  (fc3): Linear(in_features=32, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normalmodel = FashionModel()\n",
    "normalmodel.apply(init_randn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom initializations\n",
    "\n",
    "We could also choose a different way to initialize the weights, where you explicitly copy some values into the weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_custom(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        rw = torch.randn(m.weight.data.size())\n",
    "        m.weight.data.copy_(rw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use this initialization scheme to implement Xavier initialization. \n",
    "\n",
    "Xavier initialization is a way of initializing the weights such that the variance of the inputs is the same as the variance of the outputs. At each layer, the fan_in and fan_out (i.e. input connections and output connections) might be different. To calculate the variance, you will multiply each weight with the inputs. Evidently, if the number of inputs is less, they will need to be multiplied with higher weights so that they can sum up to the product of a larger number of outputs with smaller weights. This is the intuition behind Xavier initialization.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_xavier(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        fan_in = m.weight.size()[1]\n",
    "        fan_out = m.weight.size()[0]\n",
    "        std = np.sqrt(2.0 / (fan_in + fan_out))\n",
    "        m.weight.data.normal_(0,std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FashionModel(\n",
       "  (fc1): Linear(in_features=784, out_features=64, bias=True)\n",
       "  (fc2): Linear(in_features=64, out_features=32, bias=True)\n",
       "  (fc3): Linear(in_features=32, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xaviermodel = FashionModel()\n",
    "xaviermodel.apply(init_xavier)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NORMAL INIT WEIGHTS\n",
      "Start Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/tf3/lib/python3.6/site-packages/ipykernel_launcher.py:11: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  # This is added back by InteractiveShellApp.init_path()\n",
      "/anaconda3/envs/tf3/lib/python3.6/site-packages/ipykernel_launcher.py:43: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1, loss: 1.30572266\n",
      "epoch: 2, loss: 0.41340820\n",
      "epoch: 3, loss: 0.24266518\n",
      "\n",
      "XAVIER INIT WEIGHTS\n",
      "Start Training...\n",
      "epoch: 1, loss: 0.00534021\n",
      "epoch: 2, loss: 0.00380051\n",
      "epoch: 3, loss: 0.00343334\n",
      "\n"
     ]
    }
   ],
   "source": [
    "### LET'S TRAIN ###\n",
    "n_epochs = 3\n",
    "\n",
    "print(\"NORMAL INIT WEIGHTS\")\n",
    "AdamOptimizer = torch.optim.Adam(normalmodel.parameters(), lr=0.001)\n",
    "normal_trainer = Trainer(normalmodel, AdamOptimizer)\n",
    "normal_trainer.run(n_epochs)\n",
    "normal_trainer.save_model('./normal_model.pt')\n",
    "print('')\n",
    "\n",
    "\n",
    "print(\"XAVIER INIT WEIGHTS\")\n",
    "AdamOptimizer = torch.optim.Adam(xaviermodel.parameters(), lr=0.001)\n",
    "xavier_trainer = Trainer(xaviermodel, AdamOptimizer)\n",
    "xavier_trainer.run(n_epochs)\n",
    "xavier_trainer.save_model('./xavier_model.pt')\n",
    "print('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XeYVOXZx/HvzdJUisIuhlAEFQsoEVgxiu+rJoZmBHsEG4oSVOw1MRoDKXbUBGOILzFGxRoVDYoxKsaCsKCiqBBExbXRUWwU7/eP52yYXbbM7s7MmfL7XNdezpxzZufew/jbs89zzn3M3RERkfzSJO4CREQk9RTuIiJ5SOEuIpKHFO4iInlI4S4ikocU7iIieUjhLnnBzIrMbJ2ZdU3ltiK5ynSeu8TBzNYlPN0a+AbYFD3/qbvflfmqGs/Mfg10dvdRcdciha1p3AVIYXL3VhWPzew94FR3f6qm7c2sqbtvzERtIvlAwzKSlczs12Z2r5lNNbPPgePNbF8zm2Vma8zsYzO72cyaRds3NTM3s27R8zuj9Y+b2edm9pKZda/vttH6IWa2yMzWmtnvzewFMxvVgJ+pl5nNjOp/3cwOSVj3YzN7K3r/cjM7L1rewcymR69ZZWbPNXSfSmFRuEs2Oxy4G2gL3AtsBM4BioEBwGDgp7W8fiRwOdAOWApMqO+2ZtYBuA+4KHrfd4H+9f1BzKw58BjwD6AEOA+418x2jjb5CzDa3VsDvYGZ0fKLgCXRa74T1ShSJ4W7ZLPn3f1Rd//W3b9y9znu/rK7b3T3JcBk4IBaXv+Au5e5+wbgLmCvBmz7Y+BVd38kWjcRWNGAn2UA0By41t03RENQjwPHRus3AD3NrLW7r3L3eQnLvwt0dff17j5zi+8sUg2Fu2SzDxKfmNluZvYPM/vEzD4DxhOOpmvyScLjL4FWNW1Yy7bfTazDwxkI5UnUXtV3gaVe+QyG94FO0ePDgWHAUjN71sz2iZZfFW33LzN7x8wuasB7SwFSuEs2q3oq15+AN4Cd3b0NcAVgaa7hY6BzxRMzMzYHcn18BHSJXl+hK/AhQPQXyTCgA2H45p5o+Wfufp67dwMOAy4xs9r+WhEBFO6SW1oDa4EvzGx3ah9vT5XHgL5mdqiZNSWM+ZfU8ZoiM2uZ8NUCeJEwZ3CBmTUzsx8AQ4H7zGwrMxtpZm2ioZ/PiU4Ljd53p+iXwtpo+abq31ZkM4W75JILgJMI4fcnwiRrWrn7p8BPgBuAlcBOwCuE8/JrcjzwVcLXQnf/BjgUGE4Ys78ZGOnui6LXnAS8Hw03jQZOiJbvCjwNrANeAG5y9+dT9gNK3tJFTCL1YGZFhCGWo9z933HXI1ITHbmL1MHMBptZ22h45XLC8MrsmMsSqZXCXaRu+xPONV9BOLf+sGiYRSRraVhGRCQP6chdRCQPxdY4rLi42Lt16xbX24uI5KS5c+eucPe6TseNL9y7detGWVlZXG8vIpKTzOz9ZLbTsIyISB5SuIuI5CGFu4hIHlK4i4jkIYW7iEgeUriLiOQhhbuISB7KuXBftgzOPRe+UWcPEZEa5Vy4z5wJN90EJ5wAm3TLAhGRauVcuB99NFx3Hdx/P4wbB+p7JiKypdjaDzTGBReE4ZlrroGSEhg/Pu6KRESyS06GO8BVV8GKFTBhQgj4s86KuyIRkeyRs+FuBn/6E6xaBWefDcXFMGJE3FWJiGSHnBtzT9S0KUydCgccACeeCE88EXdFIiLZIafDHaBlS3jkEdhjDzjySHjppbgrEhGJX86HO0DbtuGovWNHOOQQWLAg7opEROKVF+EOsP328OST0KIFDBoE7yfVzl5EJD/lTbgD7LgjzJgB69bBwIGwfHncFYmIxCOvwh2gd2947DFYuhSGDoXPP4+7IhGRzMu7cAfYf/9wBesrr8Bhh6kPjYgUnrwMd4Af/ximTIGnn4bjjlMfGhEpLHkb7hDOfb/+enjwQTjzTPWhEZHCkbNXqCbr/PPDxOpVV4U2BRMmxF2RiEj65X24A/z2tyHgf/3rEPBnnx13RSIi6VUQ4W4Gt94KK1fCOedA+/ZhHF5EJF/VOeZuZlPMbJmZvVHD+uPMbH709aKZfS/1ZTZeRR+aAw+EUaPg8cfjrkhEJH2SmVC9HRhcy/p3gQPcvTcwAZicgrrSoqIPzZ57qg+NiOS3OsPd3Z8DVtWy/kV3Xx09nQV0TlFtadGmTThq79RJfWhEJH+l+lTI0UCNAx5mNsbMysysbHmMvQEq+tC0bBnaFLz3XmyliIikRcrC3cwOIoT7JTVt4+6T3b3U3UtLSkpS9dYN0r176EPz5Zch4Jcti7UcEZGUSkm4m1lv4DZguLuvTMX3zIQ99wx9aMrLYcgQ+OyzuCsSEUmNRoe7mXUF/g6c4O6LGl9SZg0YEPrQvPZa6EPz9ddxVyQi0njJnAo5FXgJ2NXMys1stJmNNbOx0SZXAO2BW8zsVTMrS2O9aXHIIXD77fDMM+pDIyL5oc6LmNy91ttOu/upwKkpqygmxx8PK1bAeefB6aeHm2+bxV2ViEjDFMQVqsk699wwsfq734U2Bb/5TdwViYg0jMK9it/8JhzB//a3IeDPPTfuikRE6k/hXoUZ/PGPoQ/NeedBcXEYshERySV53c+9oYqK4K674KCD4OSTYfr0uCsSEakfhXsNWraEhx8O92Q96ih44YW4KxIRSZ7CvRYVfWg6dw637Xv99bgrEhFJjsK9Dh06hD40W28NgwapD42I5AaFexK6dQt9aL7+Gn70I/WhEZHsp3BP0h57hD40H34IgwerD42IZDeFez3stx88+GAYex8+XH1oRCR7KdzraciQ0Ifm2Wdh5EjYuDHuikREtqRwb4DjjoMbb4SHHgp9aNzjrkhEpDJdodpA55wDy5eHdgUlJaFdgYhItlC4N8KECaEPTUWjsfPOi7siEZFA4d4IZjBpUgj488+H9u3hxBPjrkpEROHeaBV9aFavhlNOgXbtwtWsIiJx0oRqCrRoEfrQ7LUXHH00PP983BWJSKFTuKdI69ahD03XrnDooepDIyLxUrinUElJ6EOzzTahD82778ZdkYgUKoV7iu2wQ+U+NJ9+GndFIlKIFO5p0KsX/OMf8PHHoQ/N2rVxVyQihUbhnib77hv60LzxhvrQiEjmKdzTaPBg+OtfYeZMGDFCfWhEJHMU7mk2ciTcfHM4VXLsWPWhEZHM0EVMGXDWWaEPzYQJUFwMV10Vd0Uiku8U7hnyq1+FOzhdfXU4ZfKCC+KuSETymcI9Qyr60KxcCRdeGI7gTzop7qpEJF8p3DOoqAjuvBPWrIHRo0MfmkMPjbsqEclHdU6omtkUM1tmZm/UsN7M7GYzW2xm882sb+rLzB8tWsDf/w59+sAxx8C//x13RSKSj5I5W+Z2YHAt64cAPaKvMcAfG19WfmvdGqZPD1ezHnoovPZa3BWJSL6pM9zd/TlgVS2bDAfu8GAWsK2ZdUxVgfmqog9Nq1bhfPglS+KuSETySSrOc+8EfJDwvDxatgUzG2NmZWZWtnz58hS8dW7r2jUE/Pr1MHAgfPJJ3BWJSL5IRbhbNcuqvVTH3Se7e6m7l5aUlKTgrXNfz57qQyMiqZeKcC8HuiQ87wx8lILvWzC+//0wybpgAQwbBl99FXdFIpLrUhHu04ATo7Nmvg+sdfePU/B9C8qgQXDHHeHsmWOPVR8aEWmcOs9zN7OpwIFAsZmVA78EmgG4+63AdGAosBj4Ejg5XcXmuxEjwkVOZ50FY8bA//1fuPhJRKS+6gx3dx9Rx3oHzkxZRQVu3LjQh2b8+HBGzdVXx12RiOQiXaGaha68ElasgGuuCQF/4YVxVyQiuUbhnoXMQpvgFSvgootCH5pRo+KuSkRyicI9SxUVhQnWVavg1FNDH5phw+KuSkRyhW7WkcVatICHHoK+fUMfmueei7siEckVCvcs16pV6EPTvbv60IhI8hTuOaC4GGbMgDZtwvnw77wTd0Uiku0U7jmiog/Nhg2hD83HukxMRGqhcM8hu+8ehmg+/TT0oVmzJu6KRCRbKdxzzD77hD40b72lPjQiUjOFew4aOBD+9jd4/nn4yU/Uh0ZEtqRwz1E/+Qn84Q/w6KNw2mng1TZZFpFCpYuYctgZZ4Q+NFdeGc6oufbauCsSkWyhcM9xV1wRAv6660IfmosvjrsiEckGCvccl9iH5pJLwhH8KafEXZWIxE3hngeaNAl9aFavDuPv7drBYYfFXZWIxEkTqnmieXN48EHYe+9wJ6eZM+OuSETipHDPI61ahZtt77hjOAf+lVfirkhE4qJwzzPt24c+NG3bhqtYFy+OuyIRiYPCPQ916RL60GzapD40IoVK4Z6ndtst9KFZtix0klQfGpHConDPY/37w8MPw9tvh17wX34Zd0UikikK9zx38MFw553wwguhZcGGDXFXJCKZoHAvAMccA5MmwWOPhfuxfvtt3BWJSLrpIqYCcfrpoU3BL38ZrmK97rpwdauI5CeFewG5/PIQ8DfcAB06hHYFIpKfFO4FxAxuuglWroRLLw1H8KNHx12ViKSDwr3ANGkCt98Oq1bBmDGhD83hh8ddlYikmiZUC1BiH5oRI+DZZ+OuSERSLalwN7PBZrbQzBab2aXVrO9qZs+Y2StmNt/Mhqa+VEmlbbap3Idm3ry4KxKRVKoz3M2sCJgEDAF6AiPMrGeVzX4B3OfufYBjgVtSXaikXvv2oU3BdtuFPjT/+U/cFYlIqiRz5N4fWOzuS9x9PXAPMLzKNg60iR63BT5KXYmSTp07h4B3D31oPtK/nEheSCbcOwEfJDwvj5YluhI43szKgenAWdV9IzMbY2ZlZla2fPnyBpQr6bDrrvD44+FuToMGhZt+iEhuSybcq7vUxas8HwHc7u6dgaHA38xsi+/t7pPdvdTdS0tKSupfraRNaSk89BAsWqQ+NCL5IJlwLwe6JDzvzJbDLqOB+wDc/SWgJVCcigIlcw4+GO66C158MbQsUB8akdyVTLjPAXqYWXcza06YMJ1WZZulwA8BzGx3Qrhr3CUHHXUU/PGP4Uya0aPVh0YkV9V5EZO7bzSzccAMoAiY4u4LzGw8UObu04ALgD+b2XmEIZtR7l516EZyxE9/GtoUXH55uIr1+uvVh0Yk1yR1haq7TydMlCYuuyLh8ZvAgNSWJnG67LIQ8BMnQkkJ/OxncVckIvWh9gNSLbMQ7CtWwM9/Ho7gTzst7qpEJFkKd6lRkybwl7+EPjRjx4aLno44Iu6qRCQZ6i0jtWreHB54APbZJ/SheeaZuCsSkWQo3KVO22wT7uK0884wfLj60IjkAoW7JKVdO5gxY3MfmkWL4q5IRGqjcJekde4M//zn5j40H34Yd0UiUhOFu9TLLrvAE0+EuzkNGhQmW0Uk+yjcpd769YNHHgktgtWHRiQ7KdylQX7wA7j7bnjppdCyQH1oRLKLwl0a7Mgj4dZbQ7vgk09WHxqRbKKLmKRRxowJbQp+8YtwFevEiepDI5INFO7SaD//eQj4m26CDh3CcxGJl8JdGs0MbrghnEFz2WXhCH7MmLirEilsCndJiSZNYMqUEPCnnx760Bx5ZNxViRQuTahKyjRrBvffH/rQjBwJTz8dd0UihUvhLilV0YemR4/Qh6asLO6KRAqTwl1SrqIPTfv2MGSI+tCIxEHhLmnRqVPoQ2MGP/qR+tCIZJrCXdKmR4/Qh2b16tBoTH1oRDJH4S5p1bdv6EOzeDEccgh88UXcFYkUBoW7pN1BB8HUqTB7duhDs3593BWJ5D+Fu2TEEUeEPjRPPKE+NCKZoIuYJGNOOw1WrAjtCYqL4cYb1YdGJF0U7pJRl14a+tBMnAglJaHhmIiknsJdMsoMrrsuBPzll4cj+LFj465KJP8o3CXjKvrQrF4NZ5wRLnY6+ui4qxLJL5pQlVg0awb33Qf77QfHHQdPPRV3RSL5ReEusdl6a3j0UdhtNzj8cPWhEUmlpMLdzAab2UIzW2xml9awzTFm9qaZLTCzu1NbpuSr7bYLp0cWF4c+NG+/HXdFIvmhznA3syJgEjAE6AmMMLOeVbbpAfwMGODuvYBz01Cr5KnvfheefDKMxQ8cCOXlcVckkvuSOXLvDyx29yXuvh64BxheZZvTgEnuvhrA3ZeltkzJdxV9aNasCQG/cmXcFYnktmTCvRPwQcLz8mhZol2AXczsBTObZWaDq/tGZjbGzMrMrGz58uUNq1jyVp8+MG0aLFmiPjQijZVMuFd3DaFXed4U6AEcCIwAbjOzbbd4kftkdy9199KSkpL61ioF4MAD4Z57YM6ccJs+9aERaZhkwr0c6JLwvDPwUTXbPOLuG9z9XWAhIexF6u2ww2Dy5HDDj1Gj1IdGpCGSCfc5QA8z625mzYFjgWlVtnkYOAjAzIoJwzRLUlmoFJbRo+F3vwvdJM85B7zq34oiUqs6r1B1941mNg6YARQBU9x9gZmNB8rcfVq0bqCZvQlsAi5yd02JSaNcckloU3DDDaEPzRVXxF2RSO4wj+mQqLS01Mt01YrU4dtvQ4vgO+6AW26B00+PuyKReJnZXHcvrWs79ZaRrNakCdx2W7hF35lnhj40xxwTd1Ui2U/tByTrNWsG994LAwbA8ceHG2+LSO0U7pITqvahmT077opEspvCXXLGttuG0yM7dIChQ+Gtt+KuSCR7Kdwlp3TsGPrQNG0KgwbBBx/U/RqRQqRwl5yz886hD83atSHg1YdGZEsKd8lJe+21uQ/N0KGwbl3cFYlkF4W75KwDDgh9aMrK1IdGpCqFu+S0ww6DP/85jMOfeCJs2hR3RSLZQRcxSc475RRYsSK0Kyguht//Hqy6XqYiBUThLnnh4otDH5rrrgt9aH75y7grEomXwl3yxjXXhCP4K68MAX/GGXFXJBIfhbvkDbMw/r5yJYwbB+3awbHHxl2VSDw0oSp5pWnT0Idm//3DBOuTT8ZdkUg8FO6Sd7baKpwD37MnHHEEvPxy3BWJZJ7CXfLSttuGq1i33159aKQwKdwlb33nO2FYplkzGDgQli6NuyKRzFG4S17baadwBP/ZZyHgV6yIuyKRzFC4S97ba6/QC/7998MQzeefx12RSPop3KUg/O//hrNo5s0Lk6zffBN3RSLppXCXgjFsWLgf61NPqQ+N5D9dxCQFZdSo0Kbg4ovDzbYnTVIfGslPCncpOBddFAL+2mtDm4Jf/SruikRST+EuBenqq8OZM+PHh4AfNy7uikRSS+EuBckMJk+GVavg7LPDEM2IEXFXJZI6mlCVgtW0KUydCv/zP2GCdcaMuCsSSR2FuxS0ij40vXqFUyRnzYq7IpHUULhLwWvbNlzF2rEjHHIIvPlm3BWJNJ7CXYTNfWiaNw9tCm65BebM0cVOkruSCnczG2xmC81ssZldWst2R5mZm1lp6koUyYwddwzj7kVFcOaZ0L8/tGkT/nvmmfDXv4ajel38JLmgzrNlzKwImAT8CCgH5pjZNHd/s8p2rYGzAXXPlpzVuze8917oIDl7djh6nz0b7rgjHM0DtG4N/frB3nuH4N97b+jaVRdDSXZJ5lTI/sBid18CYGb3AMOBqiOTE4BrgAtTWqFIhpnBDjuEr6OPDss2bYKFCzcH/pw5cOONsGFDWN+hQwj5xMAvLo7vZxBJJtw7AR8kPC8H9kncwMz6AF3c/TEzqzHczWwMMAaga9eu9a9WJCZFReHOTj17hhYGEMbj58+vHPjTp4N7WN+9e+XA79sXWrWK7UeQApNMuFf3x6b/d6VZE2AiMKqub+Tuk4HJAKWlpV7H5iJZrUWLzeFd4bPPQufJisB/+WW4776wrkmT8Msh8eh+zz3DJK5IqiUT7uVAl4TnnYGPEp63BvYAnrUw6PgdYJqZDXP3slQVKpIL2rSBAw8MXxWWLds8dj9nTjiv/i9/CetatAj95hMDf5ddwi8CkcYw99oPoM2sKbAI+CHwITAHGOnuC2rY/lngwrqCvbS01MvKlP1SeNzDpG1i4M+dC198Eda3aQOlpZUDv3NnTdhKYGZz3b3OMxLrPHJ3941mNg6YARQBU9x9gZmNB8rcfVrjyxUpHGZhPL57dzjmmLBs06ZwE+/EwL/+eti4MazffvvNQd+/fwj/9u3j+xkk+9V55J4uOnIXqd3XX8Nrr1UO/Lff3rx+p50qT9j26QPbbBNfvZIZKTtyF5F4tGwJ++wTviqsXRuGcCrC/oUX4J57wromTWCPPSoH/h57QLNm8dQv8dKRu0iO++STykf3c+aEVsYQfkH06VM58HfeWRO2uSzZI3eFu0iecYclSyoH/rx58OWXYX3btltecNWpU7w1S/IU7iLyXxs3hr44iYE/f/7mPjkdO245YbvddvHWLNVTuItIrb76Cl59tXLgL1q0ef3OO1cO/D59Qv97iZcmVEWkVlttBfvuG74qrFkDZWWbA3/mTLj77rCuqChM0CYGfq9e4Y5Wkn105C4itfroo80TtRVH+GvWhHVbbRV65iSO4e+0ky64SicNy4hIWrjD4sWVA3/evHBePoSx+oqwrwj8jh3jrTmfKNxFJGM2bIAFCyoH/htvbJ6w7dRp83DO3nuHCdttt4235lylcBeRWH35JbzySuXAX7x48/pddqkc+HvtpQnbZGhCVURitfXWMGBA+KqwalXlCdt//QvuvDOsa9o0tEBOnLDdfXdN2DaUjtxFJFYfflj5loZlZaHNAoRfEH37Vg787t0Le8JWwzIikpO+/TYM3yQG/iuvhDtfQeiGWXXCdvvt4605kzQsIyI5qUmTMB6/yy5w/PFh2YYNYYI2MfCffDL8IgDo0qXy0X2/fqEvfiHTkbuI5KQvvginYCZO2C5ZEtaZwa67Vg78730v3Pkq12lYRkQKzsqVm8O+IvA//TSsa9YMeveuHPi77RauvM0lCncRKXjuUF5euR3ynDnw+edhfatWW07Y7rBDdk/YasxdRAqeWRiP79IFjjwyLPv229AgLbGdws03w/r1YX1xceXz7/feGzp0iO9naCiFu4gUlCZNwnDMbrvBCSeEZevXw+uvVz7Cf/zxcOQP4Wg+MfD79YPWreP7GZKhYRkRkWqsWxcmbBMD/913wzqzcIFVYuD37p2ZCVuNuYuIpNjy5eEiq8RTMpcvD+uaNw9n5CSO3++6a+pvaahwFxFJM3dYurTy+H1ZWTjqhzB0069f5cDv0qVxE7aaUBURSTOzMB6/ww5w1FFh2aZNsHBh5cCfODFciAVhcvaSS+D889Nbm8JdRCSFioqgZ8/wddJJYdk334R71lYEfib622tYRkQkhyQ7LJPioX4REckGCncRkTykcBcRyUNJhbuZDTazhWa22MwurWb9+Wb2ppnNN7N/mdkOqS9VRESSVWe4m1kRMAkYAvQERphZzyqbvQKUuntv4AHgmlQXKiIiyUvmyL0/sNjdl7j7euAeYHjiBu7+jLt/GT2dBXRObZkiIlIfyYR7J+CDhOfl0bKajAYer26FmY0xszIzK1tecc2uiIikXDLhXt2FstWeHG9mxwOlwLXVrXf3ye5e6u6lJSUlyVcpIiL1kswVquVAl4TnnYGPqm5kZgcDlwEHuPs3dX3TuXPnrjCz95MttIpiYEUDX5tO2VoXZG9tqqt+VFf95GNdSZ2wUucVqmbWFFgE/BD4EJgDjHT3BQnb9CFMpA529/80sOCkmVlZMldoZVq21gXZW5vqqh/VVT+FXFedwzLuvhEYB8wA3gLuc/cFZjbezIZFm10LtALuN7NXzWxa2ioWEZE6JdU4zN2nA9OrLLsi4fHBKa5LREQaIVevUJ0cdwE1yNa6IHtrU131o7rqp2Driq0rpIiIpE+uHrmLiEgtFO4iInko68I9iSZlLczs3mj9y2bWLWHdz6LlC81sUIbrqrF5mpltis4iSvmZREnUNcrMlie8/6kJ604ys/9EXydluK6JCTUtMrM1CevSub+mmNkyM3ujhvVmZjdHdc83s74J69K5v+qq67ionvlm9qKZfS9h3Xtm9nq0v1J6B5wk6jrQzNYm/HtdkbCu1s9Amuu6KKGmN6LPVLtoXVr2l5l1MbNnzOwtM1tgZudUs03mPl/unjVfQBHwDrAj0Bx4DehZZZszgFujx8cC90aPe0bbtwC6R9+nKIN1HQRsHT0+vaKu6Pm6GPfXKOAP1by2HbAk+u920ePtMlVXle3PAqake39F3/t/gb7AGzWsH0pon2HA94GX072/kqxrv4r3IzTxezlh3XtAcUz760DgscZ+BlJdV5VtDwWeTvf+AjoCfaPHrQnXB1X9/zFjn69sO3Kvs0lZ9Pyv0eMHgB+amUXL73H3b9z9XWBx9P0yUpfH0zwtmf1Vk0HAP919lbuvBv4JDI6prhHA1BS9d63c/TlgVS2bDAfu8GAWsK2ZdSS9+6vOutz9xeh9IYPN+ZLYXzVpzGcz1XVl5PPl7h+7+7zo8eeE64Kq9uHK2Ocr28I9mSZl/93GwwVWa4H2Sb42nXUlqto8raWFhmmzzOywFNVUn7qOjP4EfMDMKlpJZMX+ioavugNPJyxO1/5KRk21p3N/1VfVz5cDT5rZXDMbE0M9+5rZa2b2uJn1ipZlxf4ys60JIflgwuK07y8Lw8V9gJerrMrY5yupi5gyKJkmZTVtk3SDswZoSPO0AxIWd3X3j8xsR+BpM3vd3d/JUF2PAlPd/RszG0v4q+cHSb42nXVVOBZ4wN03JSxL1/5KRhyfr6SZ2UGEcN8/YfGAaH91AP5pZm9HR7aZMA/Ywd3XmdlQ4GGgB1myvwhDMi+4e+JRflr3l5m1IvwyOdfdP6u6upqXpOXzlW1H7sk0KfvvNhb63rQl/HmWVIOzNNaV2DxtmCc0T3P3j6L/LgGeJfxGz0hd7r4yoZY/A/2SfW0660pwLFX+ZE7j/kpGTbWnc38lxcx6A7cBw919ZcXyhP21DHiI1A1H1sndP3P3ddHj6UAzMysmC/ZXpLbPV8r3l5k1IwT7Xe7+92o2ydznK9WTCo2ckGhKmEjozuZJmF5VtjmTyhOq90WPe1F5QnUJqZtQTaauPoQJpB5Vlm8HtIgeFwP/IUUTS0mrrgVvAAABU0lEQVTW1THh8eHALN88gfNuVN920eN2maor2m5XwuSWZWJ/JbxHN2qeIDyEyhNes9O9v5KsqythHmm/Ksu3AVonPH6R0MAvU3V9p+LfjxCSS6N9l9RnIF11ResrDvy2ycT+in7uO4Aba9kmY5+vlO3oFP6DDSXMMr8DXBYtG084GgZoCdwffdBnAzsmvPay6HULgSEZrusp4FPg1ehrWrR8P+D16MP9OjA6w3X9DlgQvf8zwG4Jrz0l2o+LgZMzWVf0/ErgqiqvS/f+mgp8DGwgHC2NBsYCY6P1Rrit5DvR+5dmaH/VVddtwOqEz1dZtHzHaF+9Fv07X5bhusYlfL5mkfDLp7rPQKbqirYZRTjJIvF1adtfhKEyB+Yn/DsNjevzpfYDIiJ5KNvG3EVEJAUU7iIieUjhLiKShxTuIiJ5SOEuIpKHFO4iInlI4S4ikof+H9VffAuIRHNaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEICAYAAABxiqLiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3X2cVWW99/HP10ExkNQEi0CCknMKevDo3JpZZqICPoCkKWaJxtEsvNM6qaC3mZp16OTBTChJKzITCDMnnwW1rFQYEh/QgyJaks+CGOkBwd/9x3WNbbbzsBlmZs3s+b5fr/2ata91rbV+e7OZ76x1rb2WIgIzM7PNtVXRBZiZWdfkADEzs1ZxgJiZWas4QMzMrFUcIGZm1ioOEDMzaxUHiFmFJNVIWitpUFv2Neuq5O+BWLWStLbkaS9gHbAxP/9iRFzV8VVtOUnfAgZGxPFF12LdW4+iCzBrLxGxXcO0pCeBf4+I+U31l9QjIjZ0RG1m1cCHsKzbkvQtSXMkXS3p78DnJO0t6R5JL0t6RtIlkrbO/XtICkmD8/Nf5Pk3Sfq7pLslDdncvnn+aEmPSloj6QeS/ijp+Fa8puGSfpfrf1DSISXzDpX0SN7+Sklfze07S7oxL7NK0u9b+55a9+IAse5uHPBLYHtgDrABOBXoC+wDjAK+2MzynwXOAd4B/BW4YHP7StoZmAucnrf7BLDn5r4QSdsA1wM3AP2ArwJzJO2au/wUmBgRfYAPA7/L7acDK/Iy78o1mrXIAWLd3R8i4rcR8UZEvBYRiyLi3ojYEBErgJnAJ5tZfl5E1EfE68BVwG6t6HsosCQirsvzpgEvtuK17ANsA/xXRLyeD9fdBIzP818HhknqExGrIuLPJe3vBgZFxPqI+N1b1mzWCAeIdXdPlT6R9H5JN0h6VtIrwPmkvYKmPFsy/SqwXVMdm+n77tI6Ip3ZsrKC2su9G/hrbHpmzF+AAXl6HDAG+KukOyXtldv/M/dbIOlxSae3YtvWDTlArLsrPw3xMuAhYNeIeDvwDUDtXMMzwMCGJ5LEP3/pb46ngV3y8g0GAX8DyHtWY4CdSYe6Zuf2VyLiqxExGDgcOFNSc3tdZoADxKxcH2AN8A9JH6D58Y+2cj2wu6TDJPUgjcH0a2GZGknbljx6An8ijeH8h6StJe0PHAzMlfQ2SZ+V9PZ8mOzv5FOa83bfl4NnTW7f2Phmzf7JAWK2qf8AJpB+wV5GGlhvVxHxHHA08N/AS8D7gPtI31tpyueA10oeyyJiHXAYMJY0hnIJ8NmIeDQvMwH4Sz40NxH4fG7/V+B2YC3wR+D7EfGHNnuBVrX8RUKzTkZSDelw1JERcVfR9Zg1xXsgZp2ApFGSts+Hos4hHYpaWHBZZs1ygJh1Dh8nfRfjRdJ3Tw7Ph6TMOi0fwjIzs1bxHoiZmbVKVV9MsW/fvjF48OCiyzAz61IWL178YkS0dCp5dQfI4MGDqa+vL7oMM7MuRdJfKunnQ1hmZtYqDhAzM2sVB4iZmbWKA8TMzFrFAWJmZq3iADEzs1ZxgJiZWas4QBrxj3/AqafC6tVFV2Jm1nk5QBqxZAn86Edw8MHw978XXY2ZWefkAGnEPvvA7NmwaBGMGQOvvVZ0RWZmnY8DpAnjxsGsWfC738ERR8D69UVXZGbWuThAmnHssXDZZXDTTfDZz8KGDUVXZGbWeThAWnDiiTBtGlxzDXzhC/DGG0VXZGbWOVT11XjbymmnpTOz/t//g969YcYMkIquysysWBXtgeT7NS+TtFzS5Ebm95Q0J8+/V9LgknlTcvsySSNL2p+U9KCkJZLqS9r/S9L/SHpA0rWSdsjtgyW9lvsvkfSjLXnhm+uss2Dy5HR21umng2/kaGbdXYt7IJJqgOnAgcBKYJGkuoh4uKTbRGB1ROwqaTwwFTha0jBgPDAceDcwX9K/RMTGvNynIuLFsk3eBkyJiA2SpgJTgDPzvMcjYrfWvdQtI8G3vw1r18JFF0GfPnDuuUVUYmbWOVSyB7InsDwiVkTEemA2MLasz1hgVp6eB4yQpNw+OyLWRcQTwPK8viZFxK0R0TBcfQ8wsLKX0v4k+P734YQT4JvfhO99r+iKzMyKU0mADACeKnm+Mrc12if/8l8D7NTCsgHcKmmxpJOa2PYXgJtKng+RdJ+k30n6RAW1t7mttoIf/xiOPjodyvrhD4uowsyseJUMojc2XFw+AtBUn+aW3Scinpa0M3CbpP+JiN+/uULpbGADcFVuegYYFBEvSdoD+I2k4RHxyiaFpDA6CWDQoEEtvLTWqamBK6+EV1+FL38ZevWCCRPaZVNmZp1WJXsgK4FdSp4PBJ5uqo+kHsD2wKrmlo2Ihp/PA9dScmhL0gTgUODYiDRcnQ+DvZSnFwOPA/9SXmxEzIyI2oio7devxXvCt9rWW8PcuTBiRDq9d968dtuUmVmnVEmALAKGShoiaRvSoHhdWZ86oOFv8COB2/Mv/jpgfD5LawgwFFgoqbekPgCSegMHAQ/l56NIg+ZjIuLVhg1I6pcH9JH03ryuFa150W1l223huutg773hmGPghhuKrMbMrGO1eAgrnw11CnALUAP8JCKWSjofqI+IOuAK4EpJy0l7HuPzskslzQUeJh2OmhQRGyW9E7g2jbPTA/hlRNycN3kp0JN0WAvgnog4GdgXOF/SBmAjcHJErGqbt6H1evdOwbH//umSJzfemKbNzKqdooq/0FBbWxv19fUtd2wDL74I++0HTz4Jt92W9krMzLoiSYsjoralfr6USRvp2zcFR//+MHo03Hdf0RWZmbUvB0gb6t8fFiyA7beHgw6Chx9ueRkzs67KAdLGBg1KIdKjBxxwADz+eNEVmZm1DwdIO9h1V5g/P91DZMQIeOqplpcxM+tqHCDtZPhwuPXWdF/1ESPg2WeLrsjMrG05QNrR7runm1H97W9w4IHw0ktFV2Rm1nYcIO3sYx+Dujp47DEYNQrWrCm6IjOztuEA6QAjRqRLnSxZAocemm5OZWbW1TlAOsihh8JVV8Gf/gTjxsH//m/RFZmZbRkHSAc66ii44or0hcOjj4bXXy+6IjOz1nOAdLDjj4dLL03jIscdBxs3triImVmnVMn9QKyNTZqUxkHOPDPdS+THP043qjIz60ocIAU544x0f/ULLoDttoOLL063zDUz6yocIAU677y0J/Lf/51C5MILi67IzKxyDpACSfC976UQ+fa3071Fzjqr6KrMzCrjACmYBDNmpBA5++y0J/KVrxRdlZlZyxwgncBWW8FPf5pC5NRT057IxIlFV2Vm1ryKzv2RNErSMknLJU1uZH5PSXPy/HslDS6ZNyW3L5M0sqT9SUkPSloiqb6k/R2SbpP0WP65Y26XpEvyuh6QtPuWvPDOpkcPuPrqdLmTE09M02ZmnVmLASKpBpgOjAaGAcdIGlbWbSKwOiJ2BaYBU/Oyw0j3Rx8OjAJm5PU1+FRE7FZ268TJwIKIGAosyM/J2x+aHycBP9ycF9oV9OwJ11wD++4Ln/88XHdd0RWZmTWtkj2QPYHlEbEiItYDs4GxZX3GArPy9DxghCTl9tkRsS4ingCW5/U1p3Rds4DDS9p/Hsk9wA6S+ldQf5fSqxf89rdQW5u+uX7rrUVXZGbWuEoCZABQekuklbmt0T4RsQFYA+zUwrIB3CppsaSTSvq8MyKeyet6Bth5M+qoCn36pMvAf+ADcPjhcNddRVdkZvZWlQRIY19viwr7NLfsPhGxO+nQ1CRJ+7ZBHUg6SVK9pPoXXnihhVV2XjvumPY+Bg2CQw6BRYuKrsjMbFOVBMhKYJeS5wOBp5vqI6kHsD2wqrllI6Lh5/PAtfzz0NZzDYem8s/nN6MOImJmRNRGRG2/fv0qeHmd1847p/ur9+0LI0fCAw8UXZGZ2T9VEiCLgKGShkjahjQoXlfWpw6YkKePBG6PiMjt4/NZWkNIA+ALJfWW1AdAUm/gIOChRtY1AbiupP24fDbWR4E1DYe6qtmAASlEevVKdzV89NGiKzIzS1oMkDymcQpwC/AIMDcilko6X9KY3O0KYCdJy4Gvkc+cioilwFzgYeBmYFJEbATeCfxB0v3AQuCGiLg5r+s/gQMlPQYcmJ8D3AisIA3E/xj48ha98i5kyBCYPx8i0s2pnnyy6IrMzEBpR6E61dbWRn19fcsdu4gHHoD99kvjI3fdBe9+d9EVmVk1krS47OsVjfJFxLuQD38Ybr4Znn8eDjgAuvA5AmZWBRwgXcyee8INN6TDWAcdBC+/XHRFZtZdOUC6oH33hWuvhaVLYfTodF8RM7OO5gDpokaOhDlz0vdDxoyB114ruiIz624cIF3YuHEwaxbceScceSSsX190RWbWnThAurhjj4XLLoMbb4TPfhY2bCi6IjPrLhwgVeDEE2HatHQl3y98Ad54o+iKzKw78A2lqsRpp6XB9HPOSTekmjEj3e3QzKy9OECqyNlnpxCZOjWFyH/9l0PEzNqPA6SKSPCd76Rb4150Ubos/LnnFl2VmVUrB0iVkeD7308h8s1vpj2Rr3+96KrMrBo5QKrQVlvBj38Mr74Kp5+eQuRLXyq6KjOrNg6QKlVTA1demULky19OIXLccUVXZWbVxKfxVrGtt4a5c9Ml4E84IZ3ma2bWVhwgVW7bbeG662DvveGYY9IXDs3M2oIDpBvo3TtdwfdDH4IjjoA77ii6IjOrBg6QbmL77eGWW+B974PDDoO77y66IjPr6ioKEEmjJC2TtFzS5Ebm95Q0J8+/V9LgknlTcvsySSPLlquRdJ+k60va7pK0JD+elvSb3L6fpDUl877R2hfdXfXtC7fdBv37p8vA33df0RWZWVfWYoBIqgGmA6OBYcAxkoaVdZsIrI6IXYFpwNS87DBgPDAcGAXMyOtrcCrpPutviohPRMRuEbEbcDfw65LZdzXMi4jzN+N1Wta/PyxYkPZIDjoIHn646IrMrKuqZA9kT2B5RKyIiPXAbGBsWZ+xwKw8PQ8YIUm5fXZErIuIJ4DleX1IGggcAlze2EYl9QH2B36zeS/JWjJoEMyfDz16pFvjPv540RWZWVdUSYAMAJ4qeb4ytzXaJyI2AGuAnVpY9mLgDKCpa8eOAxZExCslbXtLul/STZKGN7aQpJMk1Uuqf8E3DW/S0KHpcNb69ek036eeankZM7NSlQRIY5fjiwr7NNou6VDg+YhY3Mx2jwGuLnn+Z+A9EfER4Ac0sWcSETMjojYiavv169fM6u2DH0wD66tXpxB59tmiKzKzrqSSAFkJ7FLyfCDwdFN9JPUAtgdWNbPsPsAYSU+SDontL+kXDZ0k7UQ61HVDQ1tEvBIRa/P0jcDWkvpWUL81Y4894Kab4G9/gwMPhJdeKroiM+sqKgmQRcBQSUMkbUMaFK8r61MHTMjTRwK3R0Tk9vH5LK0hwFBgYURMiYiBETE4r+/2iPhcyfo+A1wfEf/b0CDpXXlcBUl75tr9664NfOxjUFcHjz0Go0bBK6+0vIyZWYsBksc0TgFuIZ0xNTcilko6X9KY3O0KYCdJy4GvAZPzskuBucDDwM3ApIjYWEFd49n08BWkYHpI0v3AJcD4HFLWBkaMgHnzYMkSOOSQdDVfM7PmqJp/B9fW1kZ9fX3RZXQpc+emS56MGAG//S307Fl0RWbW0SQtjojalvr5m+i2iaOOgiuuSGdoHX00vP560RWZWWflALG3OP54uPTSdBHGCRNgYyUHHc2s2/H9QKxRkyalcZAzz4RevWDmzHSjKjOzBg4Qa9IZZ8DatXDBBemKvhdfnG6Za2YGDhBrwXnnpRCZNg222w4uvLDoisyss3CAWLMkuOiidDjr299OeyJnnVV0VWbWGThArEUSzJiRQuTss9OeyFe+UnRVZlY0B4hVpKYGfvYzePVVOPXUtCcycWLRVZlZkXxejVWsRw+4+up0uZMTT0zTZtZ9OUBss/TsCddcA/vuC5//fPquiJl1Tw4Q22y9eqXLnNTWpm+u33Zb0RWZWREcINYqffqky8B/4AMwdizcdVfRFZlZR3OAWKvtuCPcemu6Re4hh4CvW2nWvThAbIvsvDMsWAB9+8LIkfDgg0VXZGYdxQFiW2zAgBQib3tbuqvho48WXZGZdQQHiLWJIUNg/nx44410L5Ennyy6IjNrbw4QazPvf386I2vt2hQiTz9ddEVm1p4qChBJoyQtk7Rc0uRG5veUNCfPv1fS4JJ5U3L7Mkkjy5arkXSfpOtL2n4m6QlJS/Jjt9wuSZfkdT0gaffWvmhrPx/5CNx8Mzz/PBxwALzwQtEVmVl7aTFAJNUA04HRwDDgGEnDyrpNBFZHxK7ANGBqXnYY6f7mw4FRwIy8vganku6zXu70iNgtP5bkttHA0Pw4CfhhZS/ROtpee8H116fDWAcdBC+/XHRFZtYeKtkD2RNYHhErImI9MBsYW9ZnLDArT88DRkhSbp8dEesi4glgeV4fkgYChwCXV1jrWODnkdwD7CCpf4XLWgf75Cfh2mth6VIYPTod1jKz6lJJgAwAnip5vjK3NdonIjYAa4CdWlj2YuAM4I1GtnlhPkw1TVLPzagDSSdJqpdU/4KPnxRq5EiYMwcWLYIxY+C114quyMzaUiUB0tg96KLCPo22SzoUeD4iFjcyfwrwfuD/AO8AztyMOoiImRFRGxG1/fr1a2QR60jjxsGsWXDnnXDkkbB+fdEVmVlbqSRAVgK7lDwfCJSfX/NmH0k9gO2BVc0suw8wRtKTpENi+0v6BUBEPJMPU60Dfko+5FVhHdYJHXssXHYZ3Hhjmt6woeiKzKwtVBIgi4ChkoZI2oY0KF5X1qcOmJCnjwRuj4jI7ePzWVpDSAPgCyNiSkQMjIjBeX23R8TnABrGNfIYyuHAQyXbOC6fjfVRYE1EPNO6l20d7cQT021x581L9xF5o7EDl2bWpbR4Q6mI2CDpFOAWoAb4SUQslXQ+UB8RdcAVwJWSlpP2PMbnZZdKmgs8DGwAJkXExhY2eZWkfqRDVkuAk3P7jcDBpIH4V4ETNu+lWtFOOy0Npp9zTroh1fTp6W6HZtY1Ke0oVKfa2tqo9xX+OpUImDIFpk6Fr38dvvtdh4hZZyNpcUTUttTPt7S1DiXBd76T9kS+9710WfhvfKPoqsysNRwg1uEkuOQS+Mc/4Nxz0+Gs//iPoqsys83lALFCbLUVXH45vPpqOpTVuzecfHLLy5lZ5+EAscLU1MCVV6YQ+dKX0q1yjzuu6KrMrFK+Gq8Vaptt4Fe/SlfvPeEEuOaaoisys0o5QKxw224L110He+8NxxyTvnBoZp2fA8Q6hd694YYb4EMfgiOOgDvuKLoiM2uJA8Q6je23h1tugfe9Dw47DO6+u+iKzKw5DhDrVPr2TXc17N8/XQb+vvuKrsjMmuIAsU6nf39YsCDtkRx0EDzS2C3HzKxwDhDrlAYNgvnzoUePdIbW448XXZGZlXOAWKc1dGg6nLV+fQqRp55qeRkz6zgOEOvUPvjBNLC+ejUccAA891zRFZlZAweIdXp77JG+G7JyJRx4IKxaVXRFZgYOEOsi9tkH6urg0Udh1Ch45ZWiKzIzB4h1GSNGpDsa3ncfHHJIupqvmRXHAWJdyqGHwlVXwZ/+BOPGwbp1RVdk1n1VFCCSRklaJmm5pMmNzO8paU6ef6+kwSXzpuT2ZZJGli1XI+k+SdeXtF2V+z4k6SeSts7t+0laI2lJfvg2RN3UUUfBFVekM7SOPhpef73oisy6pxYDRFINMB0YDQwDjpE0rKzbRGB1ROwKTAOm5mWHke6PPhwYBczI62twKlD+NbGrgPcDHwLeBvx7yby7ImK3/Di/spdo1ej44+HSS9NFGCdMgI0bi67IrPupZA9kT2B5RKyIiPXAbGBsWZ+xwKw8PQ8YIUm5fXZErIuIJ4DleX1IGggcAlxeuqKIuDEyYCEwsHUvzardpEnp3upXXw1f/CK88UbRFZl1L5UEyACg9CtcK3Nbo30iYgOwBtiphWUvBs4AGv1vnw9dfR64uaR5b0n3S7pJ0vAmljtJUr2k+hdeeKGCl2dd2RlnwDnnpENaX/0qRBRdkVn3UckdCdVIW/l/06b6NNou6VDg+YhYLGm/JrY7A/h9RNyVn/8ZeE9ErJV0MPAbYOhbVh4xE5gJUFtb618n3cB558HatTBtGvTpA9/6VtEVmXUPlQTISmCXkucDgaeb6LNSUg9ge2BVM8uOAcbkINgWeLukX0TE5wAknQv0A77YsGBEvFIyfaOkGZL6RsSLFb1Sq1oSXHRROq33wgvTvUWmTCm6KrPqV8khrEXAUElDJG1DGhSvK+tTB0zI00cCt+cxjDpgfD5Lawhpj2FhREyJiIERMTiv7/aS8Ph3YCRwTES8eXhL0rvyuAqS9sy1v9SqV21VR4IZM+DYY+Gss+AHPyi6IrPq1+IeSERskHQKcAtQA/wkIpZKOh+oj4g64ArgSknLSXse4/OySyXNBR4GNgCTIqKl82V+BPwFuDvnxa/zGVdHAl+StAF4DRifQ8oMgJoa+NnP4NVX4StfSXsiX/hC0VWZVS9V8+/g2traqK+vL7oM62Dr1sHYsXDrrfDLX8L48UVXZNa1SFocEbUt9fM30a3q9OwJv/41fOIT8PnPp2tomVnbc4BYVerVC66/HnbfHT7zmfStdTNrWw4Qq1p9+sDNN8MHPpAOad11V8vLmFnlHCBW1XbcMY2FDBqUruDrITGztuMAsaq3886wYAH07QsjR8KDDxZdkVl1cIBYtzBgQAqRt70t3dXw0UeLrsis63OAWLcxZAjMn58uujhiBDz5ZNEVmXVtDhDrVt7//nRG1tq1cMAB8HT5RXnMrGIOEOt2PvKRdHbWc8+lw1m+aLNZ6zhArFvaa6/0PZEVK9LA+ssvF12RWdfjALFu65OfhGuvhYcegoMPToe1zKxyDhDr1kaNgjlzYOFCGDMGXnut6IrMug4HiHV748bBrFlw553psifr1xddkVnX4AAxI91H5LLL4IYb0vSGDUVXZNb5VXJHQrNu4cQT010Nv/rVdDHGn/4UtvKfWGZNcoCYlTjttDSYfs456YZU06enux2a2Vs5QMzKnH12CpGpU1OIfPe7DhGzxlS0gy5plKRlkpZLmtzI/J6S5uT590oaXDJvSm5fJmlk2XI1ku6TdH1J25C8jsfyOrdpaRtmbUmC73wHJk2C730PLrig6IrMOqcWA0RSDTAdGA0MA46RNKys20RgdUTsCkwDpuZlh5Hujz4cGAXMyOtrcCrwSNm6pgLTImIosDqvu8ltmLUHCS65BI4/Hs49Fy66qOiKzDqfSvZA9gSWR8SKiFgPzAbGlvUZC8zK0/OAEZKU22dHxLqIeAJYnteHpIHAIcDlDSvJy+yf10Fe5+EtbMOsXWy1FVx+ORx1FHz96/CjHxVdkVnnUkmADACeKnm+Mrc12iciNgBrgJ1aWPZi4AzgjZL5OwEv53WU929qG5uQdJKkekn1L/giR7aFamrgyivh0EPhy19O02aWVBIgjf2VHxX2abRd0qHA8xGxeDO2VUkdRMTMiKiNiNp+/fo1sojZ5tlmG/jVr+BTn0qHtK65puiKzDqHSgJkJbBLyfOBQPlFsN/sI6kHsD2wqpll9wHGSHqSdEhsf0m/AF4EdsjrKN9WU9swa3fbbgvXXQcf/SgccwzcdFPRFZkVr5IAWQQMzWdHbUMaFK8r61MHTMjTRwK3R0Tk9vH5DKohwFBgYURMiYiBETE4r+/2iPhcXuaOvA7yOq9rYRtmHWK77dI31T/0Ifj0p9OlT8y6sxYDJI83nALcQjpjam5ELJV0vqQxudsVwE6SlgNfAybnZZcCc4GHgZuBSRGxsYVNngl8La9rp7zuJrdh1pF22AFuuQXe9740LnLPPUVXZFYcVfMf8bW1tVFfX190GVaFnnkG9t0XXnwR7rgDdtut6IrM2o6kxRFR21I/X+nHrBX694cFC+Dtb093NXyk/NtMZt2AA8SslQYNgvnzoUcPGDECHn+86IrMOpYDxGwLDB0Kt92W7iEyYgQ89VTLy5hVCweI2Rb64AfTwPrq1XDAAfDcc0VXZNYxHCBmbWCPPeDGG2HlyjQmssrfULJuwAFi1kb22Qfq6uDRR9O91l95peiKzNqXA8SsDY0YkS57ct996Xsir75adEVm7ccBYtbGDjsMfvEL+OMfYdw4WLeu6IrM2ocDxKwdHH10uhT8rbem6ddfL7ois7bnADFrJyecAD/4QboI4/HHw8aWLuJj1sX4nuhm7eiUU+Af/4DJk6FXL5g50/dXt+rhADFrZ2eemULkggugd2+YNs0hYtXBAWLWAc47D9auTeGx3XbwrW8VXZHZlnOAmHUACS66KO2JXHhh2hOZMqXoqsy2jAPErINIMGNGCpGzzkpX891/f9hvP6itTbfONetKHCBmHaimBn72Mxg8OH1r/eyzU3uvXumb7Pvt50CxrsM3lDIr0Isvwu9/n26Pe+ed8OCDqd2BYkWq9IZSFQWIpFHA94Ea4PKI+M+y+T2BnwN7AC8BR0fEk3neFGAisBH4SkTcImlb4PdAT9Je0LyIODf3vwvok1e9M+ke6odL2o90f/Qn8rxfR8T5zdXtALGuprlA+fjHNw2Urbcurk6rbpUGSIuHsCTVANOBA4GVwCJJdRHxcEm3icDqiNhV0nhgKnC0pGHAeGA48G5gvqR/AdYB+0fEWklbA3+QdFNE3BMRnyjZ9jWk0GhwV0Qc2lLNZl1V377w6U+nB7w1UM46K7U7UKwzqGQMZE9geUSsAJA0GxgLlAbIWOCbeXoecKkk5fbZEbEOeELScmDPiLgbWJv7b50fm+wKSeoD7A+c0IrXZVYVHCjWmVUSIAOA0vusrQT2aqpPRGyQtAbYKbffU7bsAHhzz2YxsCswPSLuLVvnOGBBRJReFHtvSfcDTwNfj4il5cVKOgk4CWDQoEEVvDyzrsOBYp1JJQHS2HdmywdOmurT5LIRsRHYTdIOwLWSPhgRD5X0Owa4vOT5n4H35MNeBwO/AYa+ZeURM4GZkMZAGn9JZtWhPFBeeKHxQOnd+62D8g4U21KVBMhQN2PaAAAKqklEQVRKYJeS5wNJewCN9VkpqQewPbCqkmUj4mVJdwKjgIcAJO1EOnQ2rqTfKyXTN0qaIalvRLxYwWsw6xb69YMjjkgPcKBY+6okQBYBQyUNAf5GGhT/bFmfOmACcDdwJHB7RISkOuCXkv6bNIg+FFgoqR/weg6PtwEHkAbeG3wGuD4i/rehQdK7gOfyevckXUn4pc1/yWbdhwPF2lOLAZLHNE4BbiGdxvuTiFgq6XygPiLqgCuAK/Mg+SpSyJD7zSUNuG8AJkXERkn9gVl5HGQrYG5EXF+y2fHAJqcKk4LpS5I2AK8B46Oav8Ri1g4cKNaW/EVCM3tTeaA8lEclHSjdS5t+kbCrcoCYbZnmAqX0LK899nCgVBMHCA4Qs7bmQOkeHCA4QMzamwOlOjlAcICYdTQHSnVwgOAAMSuaA6VrcoDgADHrbJ5/ftNAWZovRuRA6VwcIDhAzDo7B0rn5ADBAWLW1ThQOgcHCA4Qs67OgVIMBwgOELNq01SgbLfdpoGy++4OlC3hAMEBYlbtHCjtwwGCA8Ssu3GgtA0HCA4Qs+7OgdI6DhAcIGa2KQdKZRwgOEDMrHkOlMY5QHCAmNnmqTRQ9tgDelRyP9cuygGCA8TMtszzz8PvfvfPQHn44dRe7YFSaYBsVeHKRklaJmm5pMmNzO8paU6ef6+kwSXzpuT2ZZJG5rZtJS2UdL+kpZLOK+n/M0lPSFqSH7vldkm6JK/rAUm7V1K7mVlr7bwzfOYzMH162ht57jmYOxeOOw7++leYPBk++lHYcUcYPRqmToV774UNG4quvGO0mJn5vuXTgQOBlcAiSXUR8XBJt4nA6ojYVdJ4YCpwtKRhpPubDwfeDcyX9C/AOmD/iFgraWvgD5Juioh78vpOj4h5ZaWMBobmx17AD/NPM7MO0RAon/lMev7cc5se8pqc/7zebjv4xCc2HUOppj2UBpW8pD2B5RGxAkDSbGAsUBogY4Fv5ul5wKWSlNtnR8Q64AlJy4E9I+JuYG3uv3V+tHQsbSzw80jH3O6RtIOk/hHxTAWvwcyszb3znc0HyplnpvZqDZRKXsIA4KmS5yt561/+b/aJiA2S1gA75fZ7ypYdAG/u2SwGdgWmR8S9Jf0ulPQNYAEwOQdQY3UMADYJEEknAScBDBo0qIKXZ2bWNrpboFRSshppK99baKpPk8tGxEZgN0k7ANdK+mBEPARMAZ4FtgFmAmcC51dYBxExMy9HbW1t9Z4hYGadXrUHSiUlrgR2KXk+EHi6iT4rJfUAtgdWVbJsRLws6U5gFPBQySGpdZJ+Cnx9M+owM+u0qi1QKjkLaxEwVNIQSduQBsXryvrUARPy9JHA7Xmsog4Yn8/SGkIaAF8oqV/e80DS24ADgP/Jz/vnnwIOBx4q2cZx+WysjwJrPP5hZl1ZQ6A0nOX17LP/PMvrL39JgbLXXuksr4MPhu9+FxYu7DxnebWYaXlM4xTgFqAG+ElELJV0PlAfEXXAFcCVeZB8FSlkyP3mkgbcNwCTImJjDolZeRxkK2BuRFyfN3mVpH6kQ1ZLgJNz+43AwcBy4FXghDZ4/WZmnUZX20PxFwnNzLqI8kBp+GJjnz5vvfTKlgSKv4mOA8TMqttzz236TflHHkntffrAiSfCRRe1br2VBkgnHJYxM7NKvPOdcNRR6QGbBkpHfIvBAWJmViXKA6W9VXQtLDMzs3IOEDMzaxUHiJmZtYoDxMzMWsUBYmZmreIAMTOzVnGAmJlZqzhAzMysVar6UiaSXgD+sgWr6Au82EbltCXXtXlc1+ZxXZunGut6T0T0a6lTVQfIlpJUX8n1YDqa69o8rmvzuK7N053r8iEsMzNrFQeImZm1igOkeTOLLqAJrmvzuK7N47o2T7ety2MgZmbWKt4DMTOzVnGAmJlZq3TLAJE0StIyScslTW5kfk9Jc/L8eyUNLpk3JbcvkzSyg+v6mqSHJT0gaYGk95TM2yhpSX7UdXBdx0t6oWT7/14yb4Kkx/JjQgfXNa2kpkclvVwyrz3fr59Iel7SQ03Ml6RLct0PSNq9ZF57vl8t1XVsrucBSX+S9JGSeU9KejC/X216n+gK6tpP0pqSf69vlMxr9jPQznWdXlLTQ/kz9Y48rz3fr10k3SHpEUlLJZ3aSJ+O+YxFRLd6ADXA48B7gW2A+4FhZX2+DPwoT48H5uTpYbl/T2BIXk9NB9b1KaBXnv5SQ135+doC36/jgUsbWfYdwIr8c8c8vWNH1VXW//8CP2nv9yuve19gd+ChJuYfDNwECPgocG97v18V1vWxhu0Boxvqys+fBPoW9H7tB1y/pZ+Btq6rrO9hwO0d9H71B3bP032ARxv5P9khn7HuuAeyJ7A8IlZExHpgNjC2rM9YYFaengeMkKTcPjsi1kXEE8DyvL4OqSsi7oiIV/PTe4CBbbTtLaqrGSOB2yJiVUSsBm4DRhVU1zHA1W207WZFxO+BVc10GQv8PJJ7gB0k9ad9368W64qIP+XtQsd9vip5v5qyJZ/Ntq6rIz9fz0TEn/P034FHgAFl3TrkM9YdA2QA8FTJ85W89c1/s09EbADWADtVuGx71lVqIukvjAbbSqqXdI+kw9uops2p64i8qzxP0i6buWx71kU+1DcEuL2kub3er0o0VXt7vl+bq/zzFcCtkhZLOqmAevaWdL+kmyQNz22d4v2S1Iv0S/iakuYOeb+UDq//G3Bv2awO+Yz1aO2CXZgaaSs/l7mpPpUs21oVr1vS54Ba4JMlzYMi4mlJ7wVul/RgRDzeQXX9Frg6ItZJOpm097Z/hcu2Z10NxgPzImJjSVt7vV+VKOLzVTFJnyIFyMdLmvfJ79fOwG2S/if/hd4R/ky6NtNaSQcDvwGG0kneL9Lhqz9GROneSru/X5K2I4XWaRHxSvnsRhZp889Yd9wDWQnsUvJ8IPB0U30k9QC2J+3KVrJse9aFpAOAs4ExEbGuoT0ins4/VwB3kv4q6ZC6IuKlklp+DOxR6bLtWVeJ8ZQdXmjH96sSTdXenu9XRSR9GLgcGBsRLzW0l7xfzwPX0naHblsUEa9ExNo8fSOwtaS+dIL3K2vu89Uu75ekrUnhcVVE/LqRLh3zGWuPQZ7O/CDtda0gHdJoGHgbXtZnEpsOos/N08PZdBB9BW03iF5JXf9GGjQcWta+I9AzT/cFHqONBhMrrKt/yfQ44J7454DdE7m+HfP0OzqqrtzvX0kDmuqI96tkG4NpelD4EDYd4FzY3u9XhXUNIo3rfaysvTfQp2T6T8CoDqzrXQ3/fqRfxH/N711Fn4H2qivPb/jjsndHvV/5tf8cuLiZPh3yGWuzN7orPUhnKDxK+mV8dm47n/RXPcC2wK/yf6aFwHtLlj07L7cMGN3Bdc0HngOW5Eddbv8Y8GD+D/QgMLGD6/oOsDRv/w7g/SXLfiG/j8uBEzqyrvz8m8B/li3X3u/X1cAzwOukv/gmAicDJ+f5Aqbnuh8Eajvo/WqprsuB1SWfr/rc/t78Xt2f/53P7uC6Tin5fN1DScA19hnoqLpyn+NJJ9aULtfe79fHSYedHij5tzq4iM+YL2ViZmat0h3HQMzMrA04QMzMrFUcIGZm1ioOEDMzaxUHiJmZtYoDxMzMWsUBYmZmrfL/AR75MoYr2FjTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "### VISUALIZATION ###\n",
    "def training_plot(metrics):\n",
    "    plt.figure(1)\n",
    "    plt.plot([m.loss for m in metrics], 'b')\n",
    "    plt.title('Training Loss')\n",
    "    plt.show()\n",
    "\n",
    "training_plot(normal_trainer.metrics)\n",
    "training_plot(xavier_trainer.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using pretrained weights\n",
    "\n",
    "In the previous section we saw that initializations can start the training from a good spot. In addition to these schemes, you might also need to have specific methods to initialize the weights in different layers. For example, you might want to use a pretrained model like Alexnet to give your network a head start for visual recognition tasks. Let's load the pretrained Alexnet model and see how it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "alexnet_model = models.alexnet(pretrained=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding Momentum\n",
    "\n",
    "We can make use of the `self.state` data-structure to maintain a copy of an accumulated gradient that we also decay at each step. Once again we use inplace operations to avoid unneccesary buffer allocation. Recall a standard update with momentum given decay rate $\\mu$.\n",
    "\n",
    "$$ \\begin{align}\n",
    "\\textbf{V'} &= \\mu \\textbf{V} - \\eta \\nabla L(\\textbf{W})\\\\\n",
    "\\textbf{W'} &= \\textbf{W} + \\textbf{V'}\\\\\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch Normalization\n",
    "\n",
    "Batch normalization is a relatively simple but significant improvement in training neural networks. In machine learning, *covariate shift* is a phenomenon in which the covariate distribution is non-stationary over the course of training. This is a common phenomenon in online learning. When training a neural network on a fixed dataset, there is no covariate shift (excluding sample noise from minibatch approximation), but the distribution of individual node and layer activity shifts as the network parameters are updated. As an abstraction, we can consider each node's activity to be a covariate of the following nodes in the network. Thus we can think of the non-stationarity of node (and layer) activations as a sort of *internal covariate shift*. \n",
    "\n",
    "Why is internal covariate shift a problem? Each subsequent layer has to account for a shifting distribution of its inputs. For saturating non-linearities the problem becomes even more dire, as the shift in activity will more likely place the unit output in the saturated region of the non-linearity.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BatchNorm(nn.Module):\n",
    "\n",
    "    def __init__(self, num_features):\n",
    "        super(BatchNorm, self).__init__()\n",
    "        self.num_features = num_features\n",
    "        self.affine = affine\n",
    "        self.weight = Parameter(torch.Tensor(num_features))\n",
    "        self.bias = Parameter(torch.Tensor(num_features))\n",
    "        self.register_buffer('running_mean', torch.zeros(num_features))\n",
    "        self.register_buffer('running_var', torch.ones(num_features))\n",
    "        self.reset_parameters()\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        self.running_mean.zero_()\n",
    "        self.running_var.fill_(1)\n",
    "        self.weight.data.uniform_()\n",
    "        self.bias.data.zero_()\n",
    "\n",
    "    def forward(self, x):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overfitting\n",
    "Deep neural networks contain multiple non-linear hidden layers and this makes them very\n",
    "expressive models that can learn very complicated relationships between their inputs and\n",
    "outputs. With limited training data, however, many of these complicated relationships\n",
    "will be the result of sampling noise, so they will exist in the training set but not in real\n",
    "test data even if it is drawn from the same distribution. This leads to overfitting and many\n",
    "methods have been developed for reducing it.\n",
    "\n",
    "## Dropout\n",
    "Dropout is a regularization technique for reducing overfitting in neural networks by preventing complex co-adaptations on training data. It is a very efficient way of performing model averaging with neural networks. The term \"dropout\" refers to dropping out units in a neural network.\n",
    "\n",
    "## Regularization (weight_decay)\n",
    "\n",
    "Weight decay specifies regularization in the neural network.\n",
    "During training, a regularization term is added to the network's loss to compute the backpropagation gradient. The weight decay value determines how dominant this regularization term will be in the gradient computation.\n",
    "\n",
    "As a rule of thumb, the more training examples you have, the weaker this term should be. The more parameters you have the higher this term should be.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FashionModel_Tricks(\n",
      "  (fc1): Linear(in_features=784, out_features=64, bias=True)\n",
      "  (bnorm1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (dp1): Dropout(p=0.2)\n",
      "  (fc2): Linear(in_features=64, out_features=32, bias=True)\n",
      "  (bnorm2): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (dp2): Dropout(p=0.1)\n",
      "  (fc3): Linear(in_features=32, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "class FashionModel_Tricks(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super(FashionModel_Tricks, self).__init__()\n",
    "        self.fc1 = nn.Linear(784, 64)\n",
    "        self.bnorm1 = nn.BatchNorm1d(64)\n",
    "        self.dp1 = nn.Dropout(p=0.2)\n",
    "        self.fc2 = nn.Linear(64, 32)\n",
    "        self.bnorm2 = nn.BatchNorm1d(32)\n",
    "        self.dp2 = nn.Dropout(p=0.1)\n",
    "        self.fc3 = nn.Linear(32, 10)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = self.dp1(self.bnorm1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.dp2(self.bnorm2(x))\n",
    "        x = F.log_softmax(self.fc3(x))\n",
    "        return x\n",
    "print(FashionModel_Tricks())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### TRAIN MODELS WITH BATCHNORM AND DROPOUT ###\n",
    "n_epochs = 10\n",
    "\n",
    "model = FashionModel_Tricks()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr = 0.001, momentum = 0.9, weight_decay = 0.001)\n",
    "btrainer = Trainer(model, optimizer)\n",
    "btrainer.run(n_epochs)\n",
    "btrainer.save_model('./dropout-batchnorm_optimized_model.pt')\n",
    "\n",
    "training_plot(vtrainer.metrics)\n",
    "training_plot(btrainer.metrics)\n",
    "\n",
    "print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Clipping\n",
    "\n",
    "During experimentation, once the gradient value grows extremely large, it causes an overflow (i.e. NaN) which is easily detectable at runtime or in a less extreme situation, the Model starts overshooting past our Minima; this issue is called the Gradient Explosion Problem.\n",
    "\n",
    "Gradient clipping will ‘clip’ the gradients or cap them to a Threshold value to prevent the gradients from getting too large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Gradient Clipping \n",
    "# `clip_grad_norm` helps prevent the exploding gradient problem. To be used before optimizer.step()during training\n",
    "torch.nn.utils.clip_grad_norm(model.parameters(), 0.25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Annealing Learning Rate\n",
    "In training deep networks, it is usually helpful to anneal the learning rate over time. Good intuition to have in mind is that with a high learning rate, the system contains too much kinetic energy and the parameter vector bounces around chaotically, unable to settle down into deeper, but narrower parts of the loss function. Knowing when to decay the learning rate can be tricky: Decay it slowly and you’ll be wasting computation bouncing around chaotically with little improvement for a long time. But decay it too aggressively and the system will cool too quickly, unable to reach the best position it can. One way of doing it is using step decay. Step decay schedule drops the learning rate by a factor every few epochs. The mathematical form of step decay is:\n",
    "\n",
    "$$\\eta = \\eta_0 * drop^{\\floor ( \\frac{epoch}{epochs\\_drop})}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def step_decay(epoch):\n",
    "    initial_lrate = 0.1\n",
    "    drop = 0.5\n",
    "    epochs_drop = 10.0\n",
    "    lrate = initial_lrate * math.pow(drop,  \n",
    "           math.floor((1+epoch)/epochs_drop))\n",
    "    return lrate"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
